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CHAPTER I 



STATEMENT OP PURPOSE AND BACKGROUND 



X . Introduction 

The development of the programed approach and the more 
recent adaptation to computer-assisted Instruction is one of 
the few educational innovations which has generated excitement 
from* both researchers and educators. The awareness, concern 
and some alarm that a segment of our population is not being 
prepared to live in or adjust to a dynamic society in the midst 
of technological innovation, coupled with the increasing recog- 
nition that schools must pace and adjust the curriculum to 
hold the potential dropout, suggests the value of a self- 
instructional and individualized process. The researchers' 
enthusiasm for this approach can be illustrated by Suppes' 

(78) emphasis to use a computer-based system to bridge the gap 
between research in learning theory and curriculum, or the 
potential of programed materials suggested by Lumsdalne (49), 
to build a science of instruction. 

It is likely, however, that the educator's continued 
Interest, in the self-instructional approach rests to a large 
extent on the realization of its potential. Although programed 
learning has encouraged a more systematic curriculum evaluation, 
the statement of behavioral objectives, and some modification 
of instructional procedures , the implementation of this 
approach has not always provided the Individualization which 
initially anticipated. A program which incorporates the 
features of individual pacing, immediate reinforcement, and 
active involvement is, not necessarily any more effective than 
traditional methods. Moreover, the manipulation of programing 
variables does not always result in a more effective presenta- 
tion of materials. Such conclusions, commonly reported in 
studies of programed learning and computer-assisted instruction, 
emphasize the need to identify more efficient ways to evaluate 
and guide in the writing, revision, and utilization of self- 
instructlonal materials. 

In the past programed and computer-assisted instructional 
materials e been developed by trial and error. For example, 
a particular piece of computer assisted instructional materials 
would be written and then tested on a typical group. The 
results would serve as feedback for further revisions of the 
programs. These revisions, in turn, would require further 
revisions. This activity continued until the program functioned 
smoothly. This trlal-and error approach is inefficient; 
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lt is both costly and time consuming. The research reported in 
this project focuses on the effective development * evaluation* 
and Individualisation of programmed materials. 



Phase I 

Efforts to develop successful programing methods have led 
to the wide variation in the application of the principles of 
learning to materials development. However* program assessment 
studies have not generated a set of principle ' to guide in the 
selection of a programing technique • Learner characteristics 
and task variables are frequently hot considered, in identifying 
the appropriate programingfthchnique . Phase I of the research 
activity responds to this need by an evaluation of four program- 
ln^ nethods which accounts fpr individual differences and 
variation in conceptual attainment. ..- r \ ' 

r ' The study did not f aypr cttgr ! parilcuUr proj^atoi^ method, 
but served as a pilot approach by seleoti^ fpur 
methods, from among, the many^echniques',, epd byrelotpigthe 
effectiveness of the method of presentation tothe c ;w 
and eel of the child and to v the conceptual level of^the behavioral 
objective . The four methods selectediJ^oye': (ly^^yelopmental 

- Skinnerian, (2) developmental - RUliEd^ tll textual counterpart 
of the Skinner type, and (4) textual counterpart of the..]RW*ECr t1 
presentation. 



Phase; 11 r 



■..Ur: 

•••r.UO-; • ■ 



Research* studies, tend to agree that reading difficulty .* 
is an important criterion for selecting textual materials ; and , 
as such* textbook selection is an exceedingly important and., 
difficult task : Reading difficulty is evpn more crucial to 
the students* success in using programed learning. The signifi- 
cance of the reading factor and the, absence of a formula adapted 
to the programed approach prompted. $he, research activity of . 
Phase n or ths reported research , ] l, The maj or objective or this 
phase of activity was the development of an automated analysis 
to manage programed materials development which Includes a 
synthesd 8 pf pptjti programing and readability variables. . 

research activity identlflf# 

ent readability and program variables which are descriptive of 
the. . framet Structure , response characteristics , and, the content 
preseptat^bh anct organlzatlon of programed; materials;. The 
automated procedures sort* locate* and, accumuiate : - a mount and 
listing of t he^ . independent variables- from the f . context of the 
programed materials . , After a printout ^^athe, variable counts 

• ... . - : >' . f il C Hj'fOX SJ1 V \r£ : 

; ; •' -.-v T f .;••• 1 in •••• l '. 
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and word listings is obtained , the i. -^pendent variables are 
related to student error rate (validated against student ability 
and achievement measures). The readability formula is described 
by an equation of the best predictors of error rate. Thus 
descriptive data are presented for all variables, those reada- 
bility variables significantly related to student performance 
are identified, and the nature of the relationship is established. 

The automated feature and related analysis offer specific 
advantages to the production of programmed materials (1) either 
directly, in the writing, revision, and refinement of materials; 
(2) or through program definition and experimentation, which 
yields an understanding of why a program modification is more 
or less effective. 

Firstly, the analysis offers direct assistance to writing 
and revislpn procedures in the following ways: (2) the automated 

feature and predicted student error rate reduce the time lag 
and expense created by field testing and manual counts; (b) the 
analysis pf the entire text allows the identification of differ- 
ences funong the writing practices of programers, the differences 
ante wigunits, hierarchies, or sequential organization; (c) the 
suimiarc ^ and analysis suggest revisions specific to the 
prq&ram context, hence avoiding the trial and error often assoc- 
iated with program modification, unrelated to student perform- 
ance hr t^e "relevant contextual variables. 

Secondly, the readability analysis relates to the experi- 
mentation in programed learning. The identification of signi- 
ficant frames , response , and content presentation and organization 
Y^labies, and the relationship of these variables to student 
perfqriAance 4 provides a comprehensive definition of program 
stbuqtufe and an evaluative model of program adaptations. The 
use of such a model avoids the ambiguities often associated with 
program descriptions, such as Skinnerian, small step, etc. 
Sliberkn&h (72) has stated the need for an adequate program des- 
cription as follows, "Ideally, it would be desirable to identify 
specific structural features of programs which invariably 
contribute tb their effectiveness . This would greatly simplify 
the evaluation of programs; unfortunately, such relationships 
have not been established . ” This concern is echoed by Glaser 
(30) who recommends a set of terms or a taxonomy for describing 
the properties of a particular program and the differences 
between, programs . " 

The need for a comprehensive and objective model of program 
structure is attested to be fragmented research findings, studies 
of isolated variables , difficulties encountered in generalizing 
beyond a given set of materials, and evaluation procedures which 
are typically restricted to changes in criterion behavior, 
neglecting an assessment of the relative influence of program 
features. The automated analysis offers a promising approach 
in program definition and evaluation. Objective counts on 








contextual data ( frame , response, content presentation and 
organization variables) c and a knowledge of the influence of 
these program features allow, the investigator to objectively 
describe l>ls materials and to relate programing , readability , 
and/or; psychological approaches to a common model of program.., 
structure. As sytch he can establish the. nature of a brt <c t hlng 
condition and distinguish genuine differences in program methods 
from those approaches in which the primary difference appears 
to be one of semantics. 
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Phase 111 t f 

The definition and evaluation of three remedial branching 
conditions derived from the- automated readability model of, 
program structure is the focus of Phase 111. The first J branch 
adaption, knowledge of, describes the lexical component, and 
is designed to reduce the negative influences of vocabulary 
difficulty and technical terminology. The second two, branch - i} 
lng conditions aref directed ; toward structural components 
sentence and topical* 
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* r . /Ih.i this study there, is an evaluation of a hierarchy 
branching conditions that encompasses what has often 4?s?cn labelled 
undert psychological learning variables »cv If has been commphly 
recognized that learning is related } to. IQ revels.- This yeans . v 
that, in learning at a certain level of abstraction or differ • 
culty, further teaching activity is required to (.make clear and 
comprehensible a piece of material to the student • With pro- 
grammed learning this goal is often accomplished by branching. 

When branching is used, the level of reading difficulty 

should go down so that - a student cap magter material that, was * 
previously too difficult for him to understand. But what, often 
happens is the reverse: the reading difficulty increases - with 
the use of branching conditions. The material becomes more 
difficult and is harder to learn. Such an occurrence is of 
course 4 self-defeating , for; branching should make, any material 
more comprehensible than It. was previously. 

There' are several, possible explanations why branching .••• 
increases the level of reading difficulty (decreased reada- 
bility). More and more detailed questions are asked in order 
to clear up a perplexing point ; Often this means that , in a • 
branching condition, the material moves from facts to.pri 
ciples and ideas to explain the facts. This stress on 
principles and ideas rather than facts may be a key factor in 
decreased readability ^because principles and facts are .usually 
frore difficult to understand than facts. Moreover, asking 
questions' -about a certain point , - even: if not at a new dif f3> 
culty level involving principles and ideas, may lessen the / ; 
readability because It may be harder for a student, to see the 
relationship of ttve new questions to the original .perplexing /; 
point. •" •"* : • - •• 
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In any event, whatever may be the factors that result In 
decreased readability, the important thing is to determine when 
the readability has. gone up or down and how it can be changed. 

■The problem is not merely to hold readability constant when 
branching is used, but rather to decrease the level of reading 
difficulty so that a student may understand the material that 
he previously... found perplexing. A fully developed readability 
model, we believe, will enable one to determine when readability 
Increases or decreases in a branching condition. And having a 
criterion for determining reading difficulty will bring together 
a number of loose ends— —the psychological variables— that have 
been felt to contribute to reading difficulty. 

Branching conditions in our program can be viewed as a 
three-fold hierarchy, with each level or branching condition 
more difficult, complex, and abstract, than the previous level. 
The first level deals with specific terms, especially mathe- 
matical and scientific, and general vocabulary. Students at 
this level acquire, in a word, knowledge of concepts. At the 
second level an analysis is made of sentence length and struc- 
ture. Relationships between concepts are reviewed. A global 
presentation or preview describes the third branching condit.icn. 



II. Developments by the Bureau of Educational Research 

The renewed interest in science education in the 1950's 
focused on the extremes in quality and quantity of the sciBnce 
being taught and the concern of many teachers who haa not 
anticipated the recent scientific 
A survey of the status of science in the Virginia 
schools, conducted by the Bureau of Educational Research in 
1963, and a follow-up study evaluating the objectivasofsclence 
teachers In Virginia combined and secondary schools identified 
areas of critical need within the state *35). The theoretical 
developments and expanding applications of programed learning 
during the same period suggested a possible approach to th. 
re-evaluation of the science curriculum and to an improved 
science instruction. 

In 1962, the Bureau of Educational Research initiated a 
series of studies to test the efficacy of 

materials at the fourth-grade level, coupled with an investi- 
gation of the influence of the individual performance by pupils 
of simple science experiments as they complete framed sections 
of the programed materials. Figure I summarizes the research 

strategy, 1962-1969. 



Stages 1 and 2, Materials Development and Program Assess- 
ment, established the feasibility and instructional effective- 
ness of the programed science materials (19) > (3b/. 



A. Research 
Functions 



B. Research 
Objectives 



C. Activities 



D. Funding 
Research 
Completed; 



Figure 1 
RESEARCH STRATEGY - 1962-69 



1. Materials Development- 
Task Analysis of 10 
Programed Science Units 
and. about 73 S cience Experiments 



3.1 Conceptual 
Development- 
Programing > 
Learning 
.Theories 



c- 



2. Program Assessment- 




3.2 Innovation- 


Upper Elementary 


c~; 


Educational 


Science 




Practice 



B 



B.3»la Establish 
the relationship 
between programing 
techniques, learner 
characteristics & 
conceptual levels 



.2a Ascertain the 
feasibility & 
instructional 
effectiveness of 
the programed 
approach - 



B»3.1b Develop a 
model of program 
. structure; reada- 
bility, program 
definition L 
experimentation 



B.2b Revise; materials. 
(1); identify research 
needs for materials 
refinement^ (31) ; 
Relate instructional 

f rocedures & student 
ehaviors to stace 
737^7 - f". 

C.3.1 Adaptation C.2 Pilot and Field 
to CAI for instruc- Studies under several 
' tional pruposes programing conditions 

(approx. 150 carried out in over 

students) & for 50 classes. 

-use as a research 
tool to store, 

- collect & analyze 
contextual & 
student data 



B.3.2a Implement 
• and demonstrate 
the programed 
approach within 
context of the 
Elementary School 

B.3«.2b Incorporate 
' program features 
within, an 
instructional 
system (Fig. 2) 



C.3.2 The training, 
supervision & 
implementation of| 
the programed- 
ceritered approach 
in 24 upper 
elementary class e; 



D.3.1 CRP# 6-1310 
Investigator: 
MacDougall; (June, 
1966-Mar ch, 1968) 
$64,940 Federal 
Support, $91,039 
Ihstit . Support . 



D.2 CRP# 1972 ] 

Investigators: Hedges 
& MacDougall; (1963- 
1965). $33,000 Federal 
Support. 

Institutional Support. 
One-year extension 
$12,000 Federal Support 



.3.2 CRP# 6-1319 
Investigator : 
MacDougall; (June, 
1966-Sept . 1967 ) . 
$33,351 Federal 
Contribution 
$17,209 Local 
Contribution 



Extension to June, 

1969, and Planned 
Research: Jacobson & 
MacDougall. Related 
Research: Computer-Simulated 
Lav/ Games - Milton Jacobson, 
Oct. 1968-Oct. 1969 
Small Grant Proposal 
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A one year's course of instruction, ten programed science 
units and approximately 75 experiments, was developed and field 
tested under several programing conditions in over 50 upper 
elementary classes < 37 ) • The reliability and validity of ten 
unit tests and a final true-false science achievement test were 
established * The instructional effectiveness of the programed 
approach was measured by science interest, attitudes, laboratory 
resourcefulness, science achievement and retention, and critical 
thinking skills. 

Although the feasibility of this approach was established, 
the investigators were not able to demonstrate that this approach 
was generally superior to conventional teaching (with supervisory 
assistance). The investigators concluded that the laboratory 
experiences contributed in large measure to the sustained 
motivation of students over one year, as well as contributing 
to their ability to solve new problems; critical thinking skills 
were likewise shown to significantly improve. The vast majority 
of students remained positively oriented toward th^ programed 
approach tfc Mughout the course of study. Student behaviors 
and related instructional procedures were identified and recorded 
for use in subsequent training and supervisory activities. 

Valuable side effects to teaching science observed during 
the field testing prompted a demonstration study to use this 
approach in the training of elementary teachers (5*0 . Stage 
3.3» Figure I, focuses on this objective. In contrast to the 
background studies in which the role of the teacher was prescribed 
to monitor the programed materials, a demonstration project was 
designed to encourage individual adaptation of the programed 
science approach within an organized instructional patter. 

Major concerns were (1) classroom organization for the teacher’s 
participation in the programed instructional process, and (2) 
the integration of classroom procedures end activities comple- 
mentary to the experiences of the children who progress at an 
exceptionally fast rate or who are retarded by low reading 
ability • 

The evaluation of the demonstration study recommended this 
approach as an effective way to (1) introduce new scientific 
content into the elementary science program, (2) Include the 
use of laboratory experiences in the science curriculum, (3) 
individualize instructional procedures, and (4) involve super- 
visory and administrative personnel in the elementary science 
program. A one year follow-up in 1967-68 of seven fully parti- 
cipating classes using several modified approaches confirmed 
the previous year's recommendations. 

The research reported in this study is summarized under 
stage 3.1, Fifcitre I. Three science units were adapted to a 
computer assisted instructional system and four programing 
techniques were pilot tested and evaluated (53) • Another 
study was carried out in 1967-68 by the Bureau of Educational 
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Research (89), (71). A program for the B5500 computer facility 
at the University of Virginia was written into which units of 
the programed science materials were read; automated counts of 
seventeen independent variables (measuring the frame structure, 
the density of technical terms, and the content and organiza- 
tion of the text) were obtained. The number of errors that 
each student made in the programed u.._ts were tabulated from 
programed materials • Error count , traditionally used as a 
measure of reading difficulty, was defined as the criterion 
variable. Error count was found to be highly related to other 
measures of achievement, including standardized tests. 

y .* ■ 

This project was designed to develop procedures of general 
applicability for measuring the reading difficulty of programed 
materials and to develop formulae directly applicable to the 
reading difficulty of fourth-grade programed science materials. 

p : ; | . j - - . r 1 *.* * \ * m * m ■ * t. ■ 

■ The materials consisted of nine fourth-grade programed 
science units which had been prepared at the Bureau of Edu- 
catlonal Research. Prom the nine units three — (1) sound, 

(2) light, and (3) heat were selected because more pupils 

completed this set of units than any other set. The programed 
science materials were used in twenty elementary schools in 
central Virginia during the school year 1966-67 . The make-up 
of the students was 440 sixth graders, 171 fifth- graders , 

.and 7B9 fourth graders. 

The independent variables were synthesized from an analysis 
v of more than 200 previous studies . in readability and an analysis 
of over 50 studies of programed learning. The variables were * 

' defined so that they could be determined automatically by a 
computer and could be related by means of a multiple regression 
equation to the reading difficulty of the programmed science 
materials. listing o.f the variables follows. The categori- 
zation lb twofold. The first eight variables (X^. - Xg) are 

the traditional reading variables. The next nine (X^ - X^), 

however* pre programming, variables, and: their isolation and 
analysis ! is ah especially important aspect of our model . The 
development of this model is presented in chapter II. .Our 
later model. Model II, expands the total: number. of. variables 
- to 25 and provides a better predictability and rationale for - 
selection than our initial study did (40), (41). The develop- 
ment and applications of model II are presented in chapters 
II and HI. 



3 

ERIC 



III. Review of Literature 



Four Program Versions 

Variables commonly classified under frame characteristics 
(step size, error rate, prompting and item position) and response 
variables (response made, type, rate and variation of reward 
and reinforcement) are components of programing techniques fre- 
quently identified to further refine and explain the efficacy of 
a programing technique (47), (48). However, a review of those 
studies investigating the influence of program variables on 
student performance suggests inconsistent findings or results 
which are not significant (31), (67), (68), ( 69 ), (72). 

One major difficulty arises in the study of the effects of 
a program variable in attempts to define and isolate the in- 
dependent variable (s) under investigation; i.e., an increase in 
step size is usually accompanied by a reduction in the number 
of reinforcements. From a summary of investigations, one can 
observe the interrelationships among the variables, which are 
suggestive of methods of programing, such as the Crowder or 
Skinner-type presentations. For example, small step, logical 
presentation is successful with knowledge of results or confirm- 
ation, while a less ordered or larger step approach may be 
compensated for by an explanation of a correct or incorrect 
response. Likewise, step size can be increased without loss in 
criterion achievement if responses are meaningful. There is 
evidence that overt responses may interfere with complete 
prompting, while prompting or cueing within limits appears 
successful. 

Because of the interrelationships among the programing 
variab?.es, and in the absence of evidence which indicates the 
power of any one programing variable, this study proposed to 
investigate methods of presentation, as opposed to the in- 
fluence of isolated variables. The two developmental tech- 
niques are based on the Skinner and RULEG systems. The textual 
counterparts of these systems incorporate features of the 
Crowder approach. 

Skinner -Holland Technique . Step characteristics can be 
described as small step, cueing, likelihood of correst response, 
and logical arrangement. Response mode and feedback are des- 
cribed by immediate feedback, overt, constructed response. Dis- 
crimination training assumes importance in the linear versions 
(75), ( 76 ). 

The RULEG System . Developed by Evans, Homme, and Glaser, 
(20), the Ruleg system of programing divides all verbal subject 
matter into two classes of statements: (1) rules to be learned, 

and (2) examples or illustrations. As in Skinner* s approach. 
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the program is ordered and the* steps are not. us rally longer 
than two sentences. A matrix is used to standardize the pre- 
sentation of basic relations among the materials to be pro- 
gramed^ and an additional matrix is constructed for each; ~ 
operator (relationship, discrimination). Rules and incomplete 
examples are at first prompted and then fading is introduced. 

The Ruleg System generally defines the basic concepts early in 
the program. As soon as a student demonstrates his ability to 
respond (by constructed response), the prompts are withdrawn. 

New information is expanded and varied. 

Two differences between the Skinner program and the Ruleg 
system are: Ruleg is more adaptable to use of negative or 

opposite examples, and movement toward a concept is rapid 
after indication of the student^ correct response. The 
successive^ approximations used in Skinner* s technique move the 
subject, slowly to concept formation. r 

Multiple choice programing . Norman Crowder in his use of 
the scrambled book, sometimes referred to as intrinsic program- 
ing, does not ‘emphasize step size, etc., but uses a student* s 
response to direct his behavior* Student responses can be 
modified dr Corrected and strengthened, where explanation of 
errors rather than prevention of error, is emphasized (14), 

(15). 

Sydney Pressey (64), (65)* (66) h&8 continued to use 
multiple choice responses, and like Crowder, does not emphasize, 
size of step ?or ordering. He disagrees with Skinner on the 
prevention of wrong answers and his work with testing instru- 
ments as instructional aids suggests that he views programed 
learning as an adjunct, rather than a replacement of, the more — 
traditional methOds of instruction. 

The four methods of presentation evaluated in Phase I are 
defined as follows : 

1. Developmental-Skinner type . The method is composed of 
a set of linear frames which follow the Skihner-Holland Technique, 
and is similar to that of Keislar, whose use of successive 
approximations and prompts suggest a likeness to the Skinner 
programs - : — r ------ 

2v Developmental-RULEG . This technique was developed by 
Evans y Homme and Glaser, and i 6 composed of a set of linear 
frames which classify verbal behavior into rules to be. learned 
(RU) and examples (EG). 

3 . Textual - prompting . . Essentially a textual counter- 
part of (l) above. 

4. Textual - explanation . Essentially a textual counter- 
part of (2) above. 
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Presentations (l) and (3) rely more heavily on prompting 
and cueing; presentations (2) and (4), rules or explanations. 
The developmental presentations require a student response 
after each frame. The textual presentations were included to 
ascertain if the use of a computer-based system requires the 
students’ responding to each frame, or can be as effectively 
used for testing, branching and monitoring. 



The evaluation of the four presentations considered 
variation in student ability levels, since there is conflicting 
evidence that a single presentation is equally effective for 
all students (7), (10), (17), (32), (55). Studies lucrative 
of the influence of difficulty levels on student performance 
(12), (21), (60), and the structural features of the four 
program versions suggested that the criterion behavior be 
classified into levels of conceptual attainment. This study 
identified the levels, knowledge, application, and relation- 
ships . 



More than 200 reading difficulty investigations have been 
made since 1800. Among the important summaries and reviews of 
previous reading studies are Gray and Leary’s (33) textbook 
which reviewed the literature up to 1935, Ctoil s (11) 

KLare's (44) articles reporting studies up to 1959 and 1903 
respectively and the 4yth Yearbook, Part II, of the National 
Society for the Study of Education and Reading Research 
Quarterly (83) which makes annual reviews of all aspects of 
reading. Although four major factors — vocabulary, sentence 
structure, idea density, and human interest— are identified 
in these studies as related to reading difficulty, most of the 
attacks on reading problems have been concentrated on vocabulary 



Perhaps the most important vocabulary studies were done by 
Thorndike when he developed his lists of 10 , 000 , 20 , 000 , and 
30,000 most common words in the English language (79) • These 
lists have been used by most instigators of vocabulary 
difficulty. Several other majox word lists were constructed 
by Buchingham and Dolch (9), and Lorge (46). The latter has 
shown frequency of words (in lists) to be positively related 
(R = .51) to the readability of passages in material. 

In 1928 Vogel and Washburne (80) created the first formula 
which related difficulty factors of written materials to specific 
reading levels. A regression equation related four factors of 
difficulty to the grade level of books. The four factors were 
the number of different words in a 1000 word sample, the number 
of words outside Thorndike’s 10,000 word list (both measured 
the vocabulary difficulty), the number of sample sentences 
(which measured the sentence structure), and the number or pre- 
positional phrases (which measured the idea density). This 
noteworthy study involved a general attack on leading difficulty. 
A decade later, in 1938, the formula was revised. 
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Johnson (42) demonstrated the relationship of reading 
levels: t a. the number of syllables in a word. In an experi- 
mental study using elementary reading material, the substi- 
tution of monosyllabic for polysyllabic words resulted in in- 
creased comprehension. 

^ . i - *• . . ’ * . 

In 1943 Plesch (22) found that the widely used Lorge and 
Gray-Leary formulas were inadequate for adults with more than 
limited average ability because the formulas did not differenti- 
ate between levels of difficulty of materials. The Flesch 
formula, a modification of the Lorge and Gray-Leary formulas, 
could be applied with greater ease and it discriminates between 
adult materials Aore effectively than the Gray -Ieary or Large 
formulas. 

Dale and Chall (1948} found that the number of words 
which were not in Dale* s 3000 word list correlated higher with 
the criterion variable (grade-level score equivalent) than 
did either the Lorge or Plesch formula, and with average 
sentence length as a second independent variable the correla- 
tion was . 70. Yoakam (82), working at . the elementary levels 
found that his readability formula and the DSstle and Lorge . 
formulas 'gave substantially the same results;. A study by 
Spache, mentioned becat of its wide usage in elementary schools 
and because it was partly validated with science books* e stab r 
lished the difficulty of books by the grade level in which . 
they were to be used. Bbrmuth (4) developed and refined a new 
technique (Cloze Procedure) for determining readability arid 
extended this work to include new independent variables from 
linguistics studies. Although his variables showed good 
relationships they are extremely difficult and laborious to 
determine. 

■r Summarizing the readability formula studies, the reading, 
difficulty factors are: 

. . • ‘ Sr- ' * . . • • "* 

1. Vocabulaityload. This may be estimated by such 
criteria as the!^hvbriber of syllables, number of 
letters per wox*d, numbear of . abstract words by 

u lists. These factors are intercorrelated . 

: ' ' . : V A r - fJ - ’ 

2. Sentence structure. This may be estimated by the 
... ... average sentence length arid the . percentage of 

: simple sentences . . . . v 

3. Idea density. This; may be estimated i j3y\tKer.ni^ber-v 

of prepositional phrases. ' ' f ^ v 

4. Human interest. This Aay be : estimated by the riuinber 
of personal references, personal pronouns, and 
personal sentences. 
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These formulas predict readability with correlations 
which range from .4 to .7. A major criticism of these formu- 
las is the widespread misapplication of them to materials for 
which their valid use had not been demonstrated. In addition, 
most of them are inadequate for determining reading levels of 
difficult or technical material. 

Readability of Science Materials 

Another basic shortcoming is that these studies were not 
necessarily related to the reading of science material. Of 
greater relevance for determining the readability difficulty 
of science materials are the studies by Pressey (63), Powers 
(62), Curtis (16) and Mallinson (51). They discovered that 
vocabulary difficulty was directly related to achievement 
in science studies. 

Pressy found out that the large technical vocabulary in 
science textbooks was an obvious source of difficulty to 
students in science courses. Powers utilized Thorndike* s 
20,000 word list to obtain a list of words in science text- 
books which were not in the first 10,000 words of Thorndike* s 
list. He developed a list which offered a starting point 
for authors of textbooks in science who sought to limit text- 
book vocabularies. 

Mallinson carried out several studies to investigate 
reading difficulty in various kinds of science textbooks. A 
follow-up study ten years later found that the reading diffi- 
culty of most textbooks is too high, supporting the initial 
findings. Herrington *s (34) study showed the undesirability 
of an indiscriminate application of the Plesch, Lorge, and 
Dale-Chall formulas to upper level materials in science. 

More recent studies by Marshall (52) and Major (50) 
attempted to determine the validity of using Flesch*s formula 
to evaluate the comprehension difficulty of physics and 
biology textbooks. Warringer identified specific factors 
which cause difficulty in physics and biology textbooks. 

Marshall *s study showed that Plesch *s Reading Ease 
Formula does not predict the difficulty of comprehension of 
high school physics textbooks. Major modified passages from 
textbooks using the Flesch formula, and found that increased 
achievement by students was due to the reduction of sentence 
length and reduced number of syllables. He found the Flesch 
formula offers a valid instrument to measure the reading diffi- 
culty of science texts and a procedure for modifying (a priori) 
these materials to increase achievement. 

Jacobson (38) compared the popularity of physics and 
chemistry texts with an experimental determination of their 
reading difficulty. The study removed the effect of the order 
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of variables that Warringer (8l) had used to develop a reading 
equation. Pour regression equations were developed which 
validly predicted the reading difficulty of physics and chemistry 
textbooks with a degree of accuracy comparable to that with 
which general material is predicted by widely used general 
readability , formulae. 

Smith and Reddens ( 77 ) concluded that -the reading diffi- 
culty of mathematics materials is too high and great diversity 
exists among materials written for the same grade levels. 

Fourth grade mathematics materials, for example, ranged from 
the third to the seventh grade level. The authors found 
considerable differences among the Fl-sch, Dale-Chall, and 

Spache formulas. 

* . * . » . . • * • 

Summarizing the results of past readability studies, one 
notes that 'vocabulary difficulty and sentence length are sig- 
nificantly related to the readability of technical materials. 
However, variables of greater significance to technical 
materials Include the use of symbols, mathematical terms, 
subject and unit differences. A valid measure of the reading 
difficulty of technical materials requires the application of 
a formula which is specifically developed for mathematical 
and scientific -subjects. 

Relationship* - of Programing Variables to Readability Analysis 

^ • • • 

The automated program and analysis of reading difficulty 
provides an objective and comprehensive definition of program 
structure by the presentation of word listings and counts on 
the variables which define frfcme structure, response character- 
istics, and content presentation and organization, and by an 
analysis of the relationships between the independent variables 
and the criterion, student error rate. This section summarizes 
the relationship between those programing and readability 
variables included^ iri the automated analysis. 

Frame Characteristics . Programing variables commonly 
associated -with a study of frame characteristics are step 
size, etfrdr. *ate, and prompting. The readability variables 
which define Frame Structure (number of paragraphs, sentences, 
words., letters, frames) offer a specific and objective defi- 
nition of step size. A comprehensive interpretation of error 
rate can be obtained from an analysis of the relationships of 
the Independent variables with the criterion, error rate, and 
the validation of error rate with achievement and ability 
measures. The variables, number of frames which contain a 
word identical to the response alternative and number of frames 
in which the same technical work appears more than once, are 
two indications of prompting. 




Response Characteristics . Programing studies of response 
characteristics often include a comparison of the effects of 
overt vs. covert responses, multiple choice vs. constructed 
response, the influence of a meaningful response, and reward. 
The response variable, per cent of response frames, identifies 
the overt-covert emphasis. The influence of a meaningful, 
response and other response modes can be observed by the. in- 
fluence of the independent variables, number of frames with 
math and scientific terms as responses, number of responses 
with yes-no, true-false response alternatives, number of one- 
three word response alternatives, and placement of response 
on student error rate. Since the programed materials adapted 
to the computer utilize only the multiple choice responses, 
the multiple choice vs. linear feature is not included in the 
readability formula; however, program modifications can be 
designed for materials with more than one response mode. 

Content Presentation and Organization . The content words 
or lexicon of the program text is represented by variables 
measuring the density of mathematical and scientific terms. 
Overlap is defined as the number of consecutive instances of 
technical word overlap. The syntax is described by three 
types of sentence structures. 

The variables of the revised version of the readability 
formula (Model II) are summarized below. 

Category I :: Frame Characteristics . Six variables which 
measure paragraph, sentence and word length, vocabulary diffi- 
culty, and number of frames per sample. 

Category II : Response Characteristics . Six variables 

which describe the number of response frames, the nature and 
relevance of the response. 

Category III : Content Presentation and Organization . 

Five variables which measure density of mathematical and 
scientific terms, overlap and repetition. 

Category IV : Sentence Structure or Syntax . Seven 

variables which identify three sentence types, the average 
number of words and technical words per sentence type. The 
remaining independent variable identifies the science unit. 

Criterion Variable . The average number of errors per 
sample . 

Branching Rationale 

The revisions of most frame and response variables hold 
a direct relationship to the readability model and a quanti- 
tative modification can be directly applied to lower the 



student error rate*;:; However^. edification^ of content pre-,. 
sentation and organization variables m^jnot be as- a^paitenfc-— r . 
or eveh posslblev? An- example of a relatively fixed variable, 
the density: of scientific , words, ma^nontribute to reading 
difficulty; although the curriculum may Strongly suggest the 
need' to"' include the -terminology. Less apparent revisions may 
also be indicated when error rate is not changed or is not 
accompanied by improved achievement. Lastly, the relationship 
with ability may suggest individual adaptations of a qualitative 
nature. 

In the absence of a taxonomy of human learning (56), the 
readability model is proposed as an evaluative model of branch- 
ing conditions. Programing techniques or branch adaptations 
can be selected to either compensate for the negative effects 
of a relatively fixed readability variable, or to make a 
contribution to individual or general performance beyond that 
accounted for by the readability formula. Those studies 
pertinent to the; development of the three branching conditions 
evaluated in this, research project are summarized below. 

Gagne (24) has emphasized sequencing of programed materials 
as an essential factor in concept acquisition and retention. 

The importance of the learner achieving success on each task 
component" 4ias been demonstrated by Gagne and his associates, 
who have analyzed learning from a “task analysis" approach 
(23). 



gilberman and Couison £?3), (74) . in. xeporting on the 
empirical development of programs in reading, arithmetic, 
Spanish and geometry, attempted to define optimal procedures 
to be used in program revision. Three principles, the "gap," 
"irrelevancies, " and "mastery" principles*., support. the "task 
analysis" of Gagne; however, Silberman and Couison accept 
both sequencing and individual adaptation to achieve mastery, 
while Gagne emphasized sequencing techniques. 

■* 1 r ** * . ' 



(2 j supports- the use of advance organizers, Or 
sorting ar^d classifying models. He urges the use of exposi- 
tory and comparative organizers in the organization of pro- 
gramed materials . n * r In .the j^fLjnaterial grganize.d along ,,; , 
parallel. Iines r , r <:he suggests that comparative organizers are 
expressly, designed to further the principle of integrative 
reconciliation; i.e., by pointing out iti ' what ways previosly 
learned, related ideas in cognitive structure are either 
basically similar to or essentially different. from_new-xonr _ 
cepts in the learning task. 



Branching is a common method to accommodate iridiV-idual 
differences*^ rli^ltial eaqper indentation by Couison ^nd: Silberman 
indicated -avaaviqg^ ef time : fbr ;the branching group, ; but no 
signif icarit- meah differences in post-test scores .!Vj- They . 

r. ) . :• _•'*{. - - . .V 1 - 
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superiority in oranching versions. Gil^n and Gargula ( 9) > 
using review branching in the Computer Ass ^ted Instruction^ 
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point out that these results are not consistent with those 
found by Skinner, Holland and Porter, Evans and Barlow. 

The programing approaches discussed under content pre 

sentation P anf o?ganiza P tion evidence individual success 

modifying student behavior. However, the present d - 
of program and organizational features do not ^earty - 
tinguish among methods, nor do empirical findings -Jeariy 
support a rationale which specifies the selection 0 £“ n 
i nQfniptional strategy or curriculum adaptation to the pro 
g?I»ed context! A sf^dy by Merrill and Stolurow (57) compared 
six preview and review treatments and found the summa y P 
condition" the most effective instructional procedure. The 

authors conclude that this finding suppo v.-i orovilal 
concept of advanced organizers and Gagne's hierarchial 
presentation. The assumptions that mastery is achievea by 
successive attainment and integration of lower lev 1 
qets and program modification is accomplished 

wi f h m loss in achievement:* qualification of tne 

principle and ?hf use of sequencing techniques may be indicated. 

Little is known about the selection and 

branching condition; although despite conflicting ev J; denc ®^ e 
the efficacy of individual adaptation to achieve ma ®^ er y 
been reported? A study (1) which finds an interaction be- 
tween the gap and mastery principles and between the irrele 
vancy and mastery principles suggests the difficu y 
fining program variables within the context of 
Some lf P ?he inconclusive results likely stem from the use 

and f^om ?Se need f or an evaluative model of program structure. 

The programing concerns related to context presentation 
and organization have much in ^ommon with a discourse ana y , 

S l iSSSSi' 

?isf Resents a discussion and analysis of the conventions a 
iriLfCmploys which have their result in lexical, structural, 
cultural and rhetorical components and constraints. 



Identify the lexicon and structure as primary expressive 
components in the presentation of information or contents A - 
brief statement of the discourse analysis which follows is 
restricted to those lexical and structural features most rele- 
vant to a programed analysis of a successful presentation of 
scientific content. 

A discussion of the lexical component distinguishes content 
words from structure (or function) words . ' The topic reflects -the. 
integration of the lexical items into a. larger form in the ..:••• 

structural environment, these words linked together by lexical 
equivalents or chains; The structural components include (a) 
sentence structure and- structural patterns; (b) structural ^ 
v/ords , e.g., words signaling a connection, constraint negation, 
causality, alternative, etc., and integrators, here, . 

which: and (c) constraints, a linguistic determinism or limitation, 
the purpose of which is to define, rather than diffuse the topic. 

■ rt'~ ■ . - V ' •*..** , ; 

In an analysis of material for first year, children, those 
recommendations which refer to the lexical and structural com- 
ponents are repetition through lexical equivalents and repitition 
of structures within a consistent and coherent design. The 
authors warn against structural ambiguities which create a : - 

cognitive gap when a structural signal Is omitted (e.g., becauf.e/- 
to reduce’ sentence length; or in the interests of simplification,! 
che use of an integrator (e.g., thir ) when its meaning is not, . 
clear. The gap, mastery and Irrelcvoncies principles are impllci 
in a repetitive lexicon and structure, and in sources of 
structural ambiguity. A dominant lexical chain and the analysis 
of constraints to direct, rather than diffuse, both earlier and 
later material, recognizes the value of sequencing and the 
notion of a preview and hierarchical design. . 

The rationale of the three branching .conditions evaluated - ; 

in Phase III of the study is presented below. 

Adaptation 1. Knowledge of - Lexi cal Component 

Scriven*s (70) conceptual description of educational 
objectives . (cognitive) is selected as a general statement of the 
tasks required of the program: . * 

• . ‘ ; .V - ' 

Knowledge of: 

a) Items of specific information included definitions of 
"terms in the field. 

b) Sequences or patterns of items of in format ion including 
sets of rules, procedures or classifications for handling 
or evaluating items of information (we ape talking about 

V mere knowledge' of the rule or classification and not - 
; the capacity to iapply it.) 
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Rationale. Readability studies have consistently shown 
that vocabulary and technical terminology contribute to reading 
difficulty and achievement. Textual adaptation of these variables 
has been successful. The readability formula includes measures 
of general vocabulary difficulty and the density of mathematical 
and scientific terms. It is assumed that revision procedures can 
offset any negative influence of general vocabulary ; however, since 
knowledge of technical terminology is often necessary to an 
understanding of a technical subject, scientific and mathematical 
terms are defined as the lexical component for which the 
curriculum w ill be adapted. 

The adaptation is designed for more effective concept 
acquisition for students with a relatively low verbal ability, 
iiarly studies, as well as later experimentation (26), indicate that 
a knowledge of terminology is most difficult to retain. The 
branch as a method of overlearning or review may improve 
retention generally or distinguish among student ability levels. 

The learning capability and condition is based on Gagne* s 
definition of the concept of the simpler type or concept by 
observation and classification (25) • 

Adaptation 2. Comprehension of - Structural Component /Sentence 

The general statement of tasks, from Scriven (70), is as 
follows : 

Comprehension and Understanding of: 

Internal relationships in the field, i.e., the way in which 
some of the knowledge claims are consequences of others and 
imply yet others, the way in which the terminology applies 
within the field; in short what might be called understanding 
of the intro- fieid syntax of the field or subfield. 

Rational^. Readability studies have commonly found sentence 
length and sentence structure significantly related to reading 
difficulty. However, where reduction in sentence length or 
complexity results in ambiguity or a cognitive gap, repetition 
of sentence structure may be recommended. Comprehension of the 
ordered relationship between two concepts is defined as the 
structural component for which the curriculum will be adapted. 

The learning capability and condition relates to Gagne* s definition 
of a principle (25) and repitition of lexical chaining in the 
discourse analysis (18). 

Adaptation 3. Comprehension of - Structural Component /Topical 

The general statement of tasks, fror Scriven (70) is 
presented as follows : 
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Comprehenslon or Understanding of: 

Application of the field or the rules, procedures, and 
aoncepts of the field. to appropriate examples, where the 
field is one that has such applications; this might be 
called the semantics of the field. 

; Rationale . Application or semantics of the field as the 
educational objective of the third adaptation is related to 
the structural component which contributes to the reader*^ 
understanding by constraining his focus and ! direction (rather 
than diffusing) the topic. (18) The constraint ofearlier asr 
well as later material is adapted to the curriculum by the 
contrast and comparison of principles. The branching condition ■ 
follows Ausubel’s use of advanced and comparative organizers (2). 



IV. Statement of Objectives 

Phase I 

The evaluation of the four program versions (. Skinner rtype, 
Ruleg, and the textual counterparts of each developmental pre- 
sent at ion) was carried out to identify the technlque(s) most 
effective for the attainment of science achievement. The 
analysis focused on the following two objectives. 

1. What is the relationship between the four programing 

versions and the science achievement of fourth and fifth 
grade students? ^ 

,2. What are the learning conditions which describe the 
relative success of each of the four presentations?. 

. The criterion, science achievement is defined by three 
levels of conceptual development, knowledge, application, and 
relationship, and total achievement. Learning conditions 
consider learner characteristics and program quality. Intelli- 
gence and sex are identified as measures of the learner. 
Differences in achievement and error among the four presentations 
and between the two programed units (heat and light) suggest 
variation in program quality. 

Phase II 

The following objectives were rspecified in the development 
and implementation of the automated readability analysis. 

1. Abstract and classify from readability and programed 
learning studiesthose independent contextual variables to be 
automatically, determined by the readability analysis.,, Validate 
the Independent variable classifications. 
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2. Develop a computer program which sorts, locates, and 
accumulates a count and testing of the independent variables 
from the context of the programed materials. 

3. Implement the readability analysis by relating the 
Independent variables to error rate. The readability formula 
is described by an equation of the best predictors of error 
rate. 



4. Apply the automated readability analysis to the 
Skinnerian and Ruleg programs (Phase I) and to the programed 
unit which incorporates the three branch conditions (Phase 
III) in order to objectively define the program structure and 
to distinguish similarities and differences between programing 
techniques. Determine if the assumptions of sequencing are 
met. 

Phase III 

A fully developed readability model requires individual 
adaptations which decrease the level of reading difficulty. 

The objectives outlines under this phase of research activity 
are as follows. 

1. Derive a branch rationale from the automated analysis. 

2. Evaluate the branching conditions: 

(1) Knowledge of - Lexical Component, (2) Comprehension of 
- Structural Component/Sentence, (3) Comprehension of - 
Structural Component/Topical . 

3. Include in the evaluation control for intelligence, 
sex, time, error rate and science achievement levels (know- 
ledge, application, relationships). 



♦ 
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CHAPTER II 

'•METHODS AND PROCEDURES 



Presented in Chapter III are the procedural steps followed in 
the eveluation of the four program versions, the development, 
application and validation of the automated readability analysis, 
and the assessment of the three branching conditions. 



I. Pour Program Versions 

Preliminary to the evaluation and analysis was the writing 
of the four program versions. 

Materials Development 

Two of the previously deyeloped programed science units, 
heat and light* were each rewritten using the four presentations: 

1. Developmental: Skinner-type 

2 . Developmental-RULEG 

3 . Textual-prompt ing 

4 . Textual-explanation 

The versions were first used on a pilot basis and then were 
adapted to an IBM 1460 computer system. The developmental branches 
(1 and 2) required the student to respond to each framed The 
summary treatments (3 and 4) were essentially a textual counterpart 
of the developmental versions. All students received the same 
subtesting series and remedial branches (if test scores indicated) 
on the computer, and all were engaged in the individual student 
performance of simple science experiments. 

Sample and Measurement 

The experimental period was initiated in September, 1967* 
Students matched by ability (Lorge-Thorndike intelligence test) 
and grade level (fourth and fifth) were randomly assigned to the 
four treatments: n s 56, heat unit; n*48, light unit. 

Learner characteristics were measured by his ability and sex. 
The time to complete the unit (in minutes), the error rate and 
subtest science achievement were gathered on each student. 




*See Boykin (6) for a study of achievement, feedback, and review 
on retention and transfer, using the sound unit. 
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In order to evaluate the level of conceptual attainment, each 
achievement Item In the sub test series was classified Into three 
levels, using Bloom’s Taxonomy (3): 

Level 1 - knowledge of corresponds to Level 1 In Bloom’s 
Taxonomy 

Level 2 - comprehension of corresponds to Levels 2 and 3 In 
Bloom's Taxonomy 

Level 3 - relationship corresponds to Levels 4 and 5 In 
Bloom's Taxonomy 

Analysis 

A multiple regression technique (5) was used to evaluate the 
relationship between the four program versions and the level of 
conceptual attainment. Learner characteristics were included in 
the analysis as predictor variables. 



II. Development of Automated Readability Analysis 

The development of the automated readability analysis was 
carried out in three major phases. First the computer program was 
written and extended. Secondly, the computer program was applied 
to the elementary programed science materials. Lastly, the 
automated analysis was validated. 

Development of Computer Program 

The procedures followed in writing the computer program were 
as follows: 

/ 

1. The content of programed science units .is analyzed by 
a computer program, developed by the Research Bureau, using 
alphabetic and numeric characters which simultaneously sort, locate, 
and accumulate content and response data. 

2. The program gives a table of all variable counts, means, 
and standard deviations, and word listings coded to page and frame. 

3. After the printout of variable ounts is obtained, a 
stepwise regression technique is used to determine the influence 
of the 25 independent variables on the criterion, error rate, and 
to obtain a multiple regression equation which gives the best pre- 
diction of error rate. 

4. The error rate was validated against intelligence and 
science achievement ( 71) . 



Ap plication of the Computer Program 

1. After the development of the computer program, the auto- 
mated technique was applied to an analysis of elementary programed 
science materials. Model I, developed In 1967 — 68, is presented as 

an initial prototype of this model. 

2. The independent variables were synthesized from ar analysis 
of rve than 200 previous studies in readability and an analysis of 
over 50 studies of programed learning. The variables were defined 
so that they could be determined automatically by a computer and 
could be related by means of a mu:. iple regression equation to the 
reading difficulty of programed science materials. The variable 
categorization of Model ± is twofold. The first eight variables 
(X x - Xg) are the traditional reading variables. The next nine 

(X 9 “ V’ however, are programing variables, and their isolation 
and analysis is an especially important aspect of our model. 

The variables defined by Model I and examples of this model 

are presented belot*. 



. MODEL I 

Traditional Reading Variables 



Average number of paragraphs per frame per sample. 

X 2 : Average number of sentences per paragraph per. sample. 

X^: Average number of words per sentence per sample. 

x 4 : Average number of Letters -petM word per sample. 

Xj-t Average number of simple sentences per sample. 

Xg : Average number of words per sample which were outside 

Thorndike*s list of 6000 words (measure of difficult words). 

Xrji Average number of mathematical and scientific words (terms) 
per sample. 

Xg: Average number of mathematical and scientific numerals or 

symbols per sample > 

Programing Variables 

X^t Percent of frames that were response frames per sample. 

^10 : P e rcent of response frames that were structures response 

frames per sample (frames which contain blanks for responses 
with a der Ignat ed number of words in answer. ) 
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X, , : Percent of response frames that were free response frames 

11 per sample (frames which require a sentence or more to answer). 

^12 : P ercen t frames that were non-response frames per sample. 

X, Percent of responses requiring mathematical or scientific 
words per sample. 

X, ,: Average number of frames using same key word or phrase 

1 consecutively per sample. 

X 1(; : Average number of words in phrase per average number of words 

in phrases in succeeding frames (measure of redundancy). 

X^g: Average number of disjoint frames per sample. 

X^: Average number of review frames per sample. 

Y, : - The criterion variable, average number of errors per sample. 

1 - 



The multiple regression equation, which gave the best prediction 
of Y (error count) with deletions of the insignificant variables, was 



Y > 0.02129989 X ? + 0.00217358 X 9 + 



0.05553026 X 1? ... .06129922 



where : X^ 



Average number of mathematical and scientific words 
(terms) per sample. 



X^ « Percent 



of frames that were response frames per sample. 




a Average 



number of review frames per sample. 



The preliminary work done in this pilot project (Model I) 
Indicates that the direction of the research is promising (71) • 

The three variables of most significance were Xy (average number 
of mathematical and scientific words or terms per sample), Xq 
( percent of frames that were response frames per sample), 
and Xjj (average number of review frames per sample). Two are 

programing variables whose relationship to reading difficulty had 
not previously been understood. Additional information obtained 
by testing would allow one to determine both the significance and 
stability of the other variables used in this pilot study, as well 
as the significance and stability of the additional aariables that 
have been incorporated in our revised program (Model II). 



As an example of Model I consider the following data consisting 
of six frames taken from a page in a computer based program system. 
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Example : MODEL I 

Data (page) 



84. The of a light wave is as fast 

as anything we know about. Some cars can 
go over a 100 miles per hour. Light 

travel faster than this. Light 

waves travel faster than cars, planes or 

sound . Light can travel in 

waves at great speed measured in 

per . Light can travel over 

186,000 miles in one . The speed 

of is 186,000 miles per second. 

This data is usually punched In natural language lormat onto 
IBM cards and input to the computer (many other inputs are permitted). 
The computer program processes the data and determines the variables 
automatically, determines a text list (indexed), a reply list 
(indexed), and a readability regression equation. 

The print-out of the program for our example data was as follows: 

AUTOMATED VARIABLE DETERMINATION 



Total number of letters 296 

Total number of words 73 

Total number of sentences 7 

Total number of paragraphs 6 

Total number of frames 6 

Number of response frames 6 

Number of structures frames 6 

Number of free response frames 0 

Number of non-response frames 0 

Number of review frames 0 

Number of disjoint frames 

Number of non-dis joint frames 2 



TEXT LIST 



A 

About 

Anything 

As 

At 

Can 

Cars 

Fast 

Faster 

Go 

Great 



Pg. No. 
0084:01 
0084:01 
0084:01 
0084:01 
0084:04 
0084:02 
0084:02 
0084:01 
0084:02 
0084:02 
0084:04 



Frame No. 
0084:02 



0084:01 

0084:02 

0084:03 

0084:03 

0084:02 




0084:04 



0084:05 
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Great 

Hour 

In 

Is 

Know 

Light 

Measured 

Miles 

Much 



Or 

Over 

Per 

Planes 

Second 

Some 

Sound 

Speed 

Than 

The 

This 

Travel 

Wave 

Waves 

We 

100 

186000 

REPLY LIST 

Light 

Miles 

Second 

Speed 

Waves 



0084:04 

0084:02 

0084:04 

0084:01 

0084:01 

0084:01 

0084:04 

0084:02 

0084:02 

0084:01 

0084:05 

0084:03 

0084:02 

0084:02 

0084:03 

0084:06 

0084:02 

0084:03 

0084:04 

0084:02 

0084:01 

0084:02 

0084:03 

0084:01 

0084:03 

0084:01 

0084:02 

0084:05 



0084:04 

0084:06 

0084:02 

0084:05 

0084:06 

0084:05 

0084:02 

0084:06 
0084:03 
0084: 06 

0084:04 

0084:04 

0084: 06 



0084:05 

0084:03 

0084:06 

0084:06 

0084:05 



0084:04 0084:05 



0084:06 

0084:05 

0084:04 

0084:01 

0084:02 



0084:05 

0084:03 



Consider a more comprehensive example which resulted from 100 
pages of programmed text input to computer program. Prom our initial 
pilot study (Model I), the following data were obtained. In this 

analysis 18,994 words and 100 pages of text were i " 

than a minute. The printout of the automated variable determination 

and the regression equation are shown below. 



AUTOMATED VARIABLE DETERMINATION (for 100 pages material) 



Total Humber of Letters 
Total Number of Words 
Total Humber of Sentences 
Total Number of Paragraphs 
Total Number of Frames. 



79926 
18994 
.t 1677 
1012 
'902 



o 
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Number of Response Frames 813 
Number of Structured Response Frames 808 
Number of Free Response Frames 4 
Number of Non-Response Frames 89 
Number of Review Frames 75 
Number of Disjoint Frames 149 
Number of Non-Dis joint Frames 753 



A regression equation was established where the variables are 
the eight traditional and nine programing variables: 



Y 



t 



» 0.04633829X 1 - 
+ -o.ooos^sxjj + 
+ 0.02764715X 7 - 

- 0.00l66l94X 10 + 

- 0.00209173X 13 + 
+ 0.0l679752X l6 + 



0.02973254X 2 + 
0.00372806X 5 + 
0.01622660X0 + 
0.00224255X 11 + 
0.01192024X l4 - 
0.05l49266X 17 - 



0.00454001X 3 

0.00616356X0 

0.00253719X 9 

0.00003604X 12 

0.00076549X 15 

0.06742650 



This regression equation related the independent variables 
to the criterion variable (error rate) with a multiple correlation 
coefficient (R ■ 0.573) comparable to those of other widely used 
reading equations cited in the literature. 

3. The later model. Model II, expands the total number of 
variables to 25 • The variables are defined below and applications 
of the automated analyais are presented in Chapter III. 



MODEL II 

The variables of the revised version of the readability 
formula (Model II) are presented below: 

Category I : Frame Characteristics 

X^: Average number of paragraphs per frame per sample. 

X 2 : Average number of sentences per paragraph per sample. 

X 3 : Average number of words per sentence per sample. 

X^: Average number of letters per word per sample. 

X,-: Average number of words per sample which were outside a 

5 standard text, i.e. Thorndike’s list of 6,000 words. This 

is a measure of difficulty words. A* dictionary ’of any 
kind can be generated to determine the frequency and 
difficulty of a word. 

X^: Average number of frames per page. 



o 
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Category II: Response Characteristics : Relevancy 

Xj : Per cent of frames that are response frames per sample. 

Xft : Per cent of frames with math and scientific terms among 
° alternative responses per sample. 

X : Per cent of frames with word in frame identical to 

9 response alternative . 

X : Per cent of frames with yes-no or true-false response 

10 alternative. 

X . : Average number of frames in which response ia placed 

11 within frame (as opposed to last word). 

X p : Per cent of frames with one word or phrase (1-3) as 

* 2 response alternative. 

Category III: Content Presentation, Organiza tion, and Overlap 

or Repltition 

Density of Mathematical and Scientific Terms 

X Average number of mathematical and scientific words 

** (terms) per sample. 

X^: Average number of letters per technical word. 

X % of frames in which the same technical term appears 

** more than once. 

X Average number of frames in which the same technical 
word appears consecutively. 

X : Average number of consecutive instances of technical 

n word overlap. 

Category IV: Sentence Structure or Syntax (Average number 

of sentences containing the following kinds of 

words ) 

X^q: Integrators: this, that, it 

X, Q : Signals: because, but, although, as since, when, then, 

19 next, consequently, however, either 

X 2Q : Comparative: than 

X- • Average number of words per sentence, X^g sentence type 

X p? : Average number of words per sentence, X^ ^20 

22 sentence type 
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%2y Average number of technical words per sentence , 

X 0 u: Average number of technical words per sentence , X, ft 
and X 2Q 19 

X 2 ^: Science unit 

Y, : The criterion variable: average number of errors per 

sample . 



The Independent variables were synthesized from an analysis 
of the probable causes of reading difficulty and a description of 
the basic kinds— literary and technical — of reading, material. 

was chosen because the paragraph Is the first major 
division of the frame and paragraphs serve the purpose 
of separating introductory material from material 
requiring a response. 

X 2 was chosen to Indicate the length of an Introduction. 

X- was chosen because average sentence length has been found 
as a variable contributing to reading difficulty 

Xjj was Included because mathematics and scientific words ten<| 
to be longer than common words. That Is* polysyllabic 
words are longer and more difficult than monosyllabic words. 

Xc was Included because previous readability studies have 

found this listing to measure general vocabulary difficulty. 

Xg. was selected as general frame of reference to step size. 



X„ - 



X 12 were selected to identify relevance of response 
structure: 



Xj is a measure of overt response; 

Xg and X^q are measures of response relevance; 

X^ and X^ 2 indicate placement and nature of response structure; 

*13 ~ *15 were selected to describe the density of mathematical 
3 3 and scientific words. These words are defined as 

having a scientific or technical meaning and are drawn 
from behavioral objectives of programed materials. 

X l6 - X, « were chosen to measure overlap or describe how closely 
* content or lexicon is linked within structure 



o 
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X_ « — X oil were chosen to describe the sentence structure and 
lo determine whether the nature of the structure creates 

ambiguity (this, that, it) or difficulty (connectors, 
comparatives), (X^g, x i 9 » *20^* or w ** ether difficulty 

results from a concomitant of sentence structure, 
technical terms and sentence length (X 2 i “ X^). 

X* identifies a biological or physical science unit 

to determine whether reading difficulty is common to 
all units. 



y was selected because it gives a measure of readability 

1 directly related to each frame and is commonly related 

to achievement measures. 



Variables X 1 - X^ (Category I) are criteria for determining the 

overall structure and complexity of any page of printed material. 

These variables point out? not only the most general features of a 
piece of reading material — number of frames per page and number of 
paragraphs per frame — but also pinpoint quite specific features such 
as the number of letters per word, number of words per sentence, ana 
number of sentences per sample. Further, since the basic atomic unit 
of any sort of discourse is a word, it is important to know whether or 
not a word is common. This information is provided by variable X^. 
Again, these variables in Category I are ways of pinpointing the * 
structure and the components of the structure of any piece or printed 
material. The remaining variables in Categories II, III, and IV 
facilitate an analysis of the properties of the components of the 
structure of any reading material. 



Of any material that is to be learned one important property 
is the kinds of responses that a student can make to questions about 
the material. Variables X ? - X 12 (Category II) provide a thorough 

and systematic account of the sorts of responses that a student can 
make. 



In analyzing any reading material a rough and general classifi- 
cation can be drawn up by labelling the material either literary 
or technical. If the material is technical, then it is important to 
specify in what way and what the relationship of one piece or section 
of technical material is to other technical material. Variables 
X,, - X,- (Category III) do yield this sort of information. An 
especially important function of these "technical variables is 
that they determine not only what are the technical words in a frame 
and their degree of complexity but also, and perhaps more importantly, 
determine the relationship of the technical words to one another in 5 
different frames, i.e., the degree of overlap. There are two 
principal reasons for wanting to find out what the degree of overlap 
is. First, the degree of repitition and reinforcement that may be 
needed can be established after overlap is measured. Secondly, 
the extent of dependency of later frames on earlier frames can be 
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spelled out if it is known how frequently technical words or terms . 
are used serially in a program. 

Finally , variables X^g - X-- In Category IV provide a 
syntactical analysis of reading^material. They enable one to 
specify how complex any sentence is In terms of relational words 
auch as pronouns , conjunctions, and adverbs. This function can 
be determined for both literary and technical material. 

4. Applications of Model II, presented In Chapter III, 

Include an (1) automated analysis of the heat unit, Skinner va. 

Ruleg, (2) an automated analysis of the heat unit which Incorporated 
three branching conditions, and (3) a check to satisfy the 
assumption of good sequential development; l.e., as students progress 
through the unit, is there an increase in the number of errors 

and the number of mathematical and scientific terms. 

5. A principle components factor analysis was completed with 
varlmax rotation (43) to validate the four categories hypothesized 
in the definition of Independent variables. 



III. Three Branching Conditions 

The three treatment affects or branching conditions are based 
on those readability variables and psychological approaches which 
have evidenced success in the promotion of learning. The treatments 
are as follows: 

Adaptation 1. Knowledge of - Lexical Component 

Objective . Knowledge of terminology is described by 
Scriven (70) and measured by Level 1 in Bloom's Taxonomy (3). 

Placement . A branch will be Included (in half the programs) 
after the presentation of each technical term. 

Learning Capability and Condition . The capability is described 
by a conceptual development which requires vocabulary and classifying 
behavior and corresponds to Qagne's concept by observation (25). 

The branch condition will Include vocabulary -reinforcement and the 
introduction of additional examples and non-examples. 

Adaptation 2. Comprehension of - Structural Component /Sentence 

Objective . Comprehension or understanding of a principle is 
described by Scriven (70) classification 2a. - application of termino- 
logy within the field or Intrafield sytax. Items will be classified 
using Levels 3 and above (application, analysis, synthesis, evaluation) 
in Bloom' 8 Taxonomy (3). 
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Placement. The development of principle which relates to two or 
more concepts concludes each hierarchy. The branching condition will 
be Introduced (in 1/3 of the programs) after the completion of the , 
hierarchy . 

Learning Capability and Condition . Gagne f s (25) definition 
of a principle which requires recall of component concepts and their 
correct ordering describes the type of human learning of ma^or 
concern to the second application. The discourse analysis (18) 
likewise recommends repltition of lexical chaining and sentence 
structure. The branch presents a review of the relationship of the 
concepts with the principle, in which the student is presented with 
additional examples of the ordered or underlying relationships and 
the sentence structure necessary to express the relationships. 

Adaptation 3. Comprehension of - Structural Component/Tcplcal 

Objective . Comprehension or understanding of a principle is 
described by Scriven's (70) classification 2c. - the semantics 
of the field, and measured by Levels 3 and above. Bloom's 
Taxonomy (3)- 

Placement . At the end of each hierarchy is a principle. 
Branching conditions will be introduced (in 1/3 of the programs) 
at the end of each hierarchy. 

Learning Capability and Condition . . The learning capability 
is the understanding of a principle. The condition, the comparison 
and contrast of a principle at the end of each hierarchy with the 
principle to be introduced in the next hierarchy describes a preview 
or a review in which a previously presented principles is integrated 
with the new material of the succeeding hierarchy. This condition 
will follow Ausubel's use of advanced and comparative organizers, 
and his principle of integrative reconciliation (2). 

The three branch adaptations were incorporated in the heat unit* 
The study was initiated^, in the fall of 1968. Forty-three fourth 
grade students were assigned to the following three treatment con- 
ditions: (1) no branching condition; (2) comprehension branches 

only - adaptations 2 or 3; and (3) knowledge and comprehension 
branches - adaptation 1 in combination with either adaptation 2 or 3« 
Time limitations precluded the number of student participants 
necessary to isolate the influence of each of the comprehension 
branches; i.e., adaptations 2 and 3 were observed as one treatment 
effort . * 



* See Moody (59) for an application of the automated analysis and 
branching conditions to two programed units included in the BSCS 
special materials. His branch conditions were found to be slgnifi 
cant and he accounted for over 8036 of the learning difficulty of 
slow learners using the automated analysis. 
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A multivariate analysis (5) was used to evaluate the three 
branching conditions in which the predictor variables were defined 
as levels of achievement, error rate, branch adaptations, time, 
intelligence and sex. The criterion variables were defined as total 
achievement and achievement at each of the three levels, knowledge, 
example and application, and relationships. 



IV. Implications for Management Design 

In s umma ry, the procedures outlined above hold promise for the 
design of a management system for self -instructional materials 
development. The relationship of the automated readability analysis 
and hierarchy of branching conditions to student direction and 
revision decisions is presented in Figure 2. The results presented 
in the following chapter suggest, with further field testing, that 
the full potential of this model can be realized. 
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Chapter III 
ANALYSIS AND FINDINGS 

Chapter III presents the analysis and findings of the 
evaluation of the four programing methods, the revised version 
of the automated analysis of program structure, and the assess- 
ment of the branching rationale in the first three sections. The 
remaining two sections of this chapter are devoted to the valida- 
tion of the readability model through factor analysis and to a 
summary of the findings. 

I. EVALUATION OF THE FOUR PROGRAMING METHODS 

This section presents the evaluation of the four programing 
methods. Two of the methods are described as developmental, 
Skinnerian and Ruleg, and two as textual. Read Skinnerian 
(prompting) and Read Ruleg (explanation). The four methods of 
presentation are defined as follows: 

1. Developmental-Skinner type . The method is composed 
of a set of linear rrames which follow the Skinner-Holland 
Technique, and is similar to that of Keislar, whose use of 
successive approximations and prompts suggest a likeness to 
the Skinner program. 

2. Developmental -RULEG . This technique was developed by 
Evaiur, Homme and Glaser, and is composed of a set of linear 
frames which classify verbal behavior into rules to be learned 
(RU) and examples (EG). 

3. Textual - prompting . Essentially a textual counterpart 
of (1) above. 

4. Textual - explanation . Essentially a textual counter- 
part of (2) above. 

The analysis focuses on the following two objectives: 

A. What is the relationship between the four programing 
versions and the science achievement of fourth and fifth grade 
students? 



B. What are the learning conditions which describe the 
relative success of each of the four presentations? 

The criterion, science achievement, is defined by three 
levels of conceptual development, knowledge, application, 
relationship, and total achievement. Learning conditions 
consider learner characteristics and program quality. Intelli 
gence and sex are identified as measures of the learner. 
Differences in achievement and error rate (error rate can only 
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be gathered for the developmental versions) between the light 
and heat units , as well as among the four presentations., suggest 
variation in program quality. 

A multiple regression technique* of the form Y ■ a,X.. + . . . 
a 25^25 + where the X's are the values c* the independent 
variables , the a*s are the regression coefficients, and the K 
is the constant for the equation, was used to test the mil 
hypotheses of the form, 

X, (where i « 1, 2, 3 ... n) does not contribute to the 
regression equation to predict the reading difficulty as 
determined by average number of errors per sample (a^ * 0). 

The F-statistic, .05 level of significance, is utilized to 
establish rejection or acceptance of the null hypotheses (ratio 
of regression means square to residual mean square). Using 
the P- ratio and the multiple R's a routine of single and 
multiple deletions establishes the combination of variables 
which best predict Y. The key to variables analyzed is pre- 
sented in Appendix A. 

variables were defined as: 

« achievement at levels 1 and 2, respectively 

* four programing versions 

* heat and light units 

* intelligence 

* sex 

* error rate 

The criterion variable was defined as science achievement 
{% of correct items on subtests), where 

Y^ » total achievement 

Yg * achievement at Level 1 - knowledge 

Y^ * achievement at Level 2 - example and application 

Yjj * achievement at Level 3 - relationship 



Predictor 

x 2 - x 3 

X 5 " X 8 
X 9 " X 10 



"11 

X 12 " X 13 



•Robert A. Bottenberg and Joseph H. Ward, Jr. Applied Multiple 
Linear Regression, Technical Documentary Report PRL-TDR-63-6 
(March, 1963). 

REGD, Adapted to B5500 computer, U. of Va., by Milton D. 
Jacobson. 
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Tbe significance of the Independent variables Is reported 
relative to the Influence of the predictor variables Included 
in each analysis. For example. If I.Q., error rate, and pro- 
gram versions are defined as the predictor variables of 
achievement , and the variable, program version. Is declared 
significant (at a given probability level), this finding Is 
Interpreted as a significant Influence of program versions 
on achievement beyond that accounted for by intelligence and 
error rate. Thus, the predictor variables control for possible 
initial differences and concomitant variation. Analyses were 
run to identify possible Interaction effects; no significant 
variation attributed to Interaction effects was observed. 

The analysis of data is presented under three sections. 
First, differences In program and unit quality are observed 
to establish If subsequent analyses will treat units and 
programs separately. Secondly, the four programing versions 
are evaluated; and lastly, the two developmental programs 
are compared. 

Unit Differences - Heat vs. Light 

1. Are there differences In achievement between the 

two units (heat and light)? 

A. Considering only those students assigned to the 
two developmental branches and controlling for Intelli- 
gence and error rate. It was found that the unit 
difference between heat and light significantly Influenced 
science achievement at all levels (See Appendix B, Table 
V). Achievement favored the light unit. 

B. The Influence of the unit difference and error 
rate on levels of science achievement was determined 
for each of the four program versions with Intelligence 
and sex controlled. The following results are summarized 
from Tables V - VIII, Appendix B. 

Developmental 

Skinnerian - unit difference (heat vs. light) does not 

Influence achievement at levels 1 and 3* 
Achievement In light Is superior at level 
2 and total. Unit differences account 
for the Influence of error rate on achieve 
ment levels. 

Ruleg - Achievement In light unit is significantly 

greater at levels 2 and 3 and total 
achievement. Error rate influences 
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total science achievement, levels 1 and 3, 
beyond that accounted for by unit diff- 
erences . 



Textual 

Read Skinner - No significant differences in achievement 
Read Ruleg were found between the two units at 

level 1 and total achievement at levels 
2 and 3 significantly favored the light 
unit . 

In summary, these findings indicate: (1) the light and 

heat units differ in quality with achievement favoring the 
light unit- (2) the influence of error rate, beyond that 
accounted for by unit differences suggests that the Ruleg 
program may be either a less effective method of presentation 
than the Skinnerian, or the writing of this unit has created 
sources of reading difficulty for the subjects and (3) the 
Skinnerian version is less influenced by unit differences 
than the Ruleg program. Unit differences are crucial at both 
comprehension levels (levels 2 and 3* application and relation- 
ships) for the textual versions. 

It can be observed from TABLE I that student attainment 
was generally more successful for the light unit. 



TABLE I 

SCIENCE ACHIEVEMENT BY TREATMENTS 
AND LEVEL OP CONCEPTUAL DEVELOPMENT 



Light Unit 





Total 


level 1 


level 2 


level 3 


Skinnerian 


95 


98 


95 


93 


Ruleg 


91 


94 


92 


90 


Read Skinnerian 


82 


84 


81 


81 


Read Ruleg 


81 


82* 


84 


78 






Heat Unit 






Skinnerian 


75 


81 


71 


78 


Ruleg 


71 


76 


69 


69 


Read Skinnerian 


65 


73 


65 


6l 


Read Ruleg 


66 


76 


63 


63 
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Because of the general superiority of the light unit, 
subsequent analyses consider the light and heat units 
separately . 

Evaluation of the Four Programed Versions 

2. Do the four program versions influence science 
achievement? 

The relationship between the four program versions and 
science achievement was analyzed controlling intelligence and 
sex. The following statements summarize the findings presented 
in TABLE IX, Appendix C. 

A. The program versions have a significant influence 
on all levels of science achievement for the light unit 
and at level 3 for the heat unit. The higher mean achieve- 
ment favors the developmental presentations (Skinner and 
Ruleg types) over the summary versions. 

B. The influence of intelligence (program and sex 
controlled) Is significant at all achievement levels for 
both units . 

C. Deletion of the sex variable indicates a significant 
Influence at level 3 achievement for the heat unit only. 

The mean achievement favors the boys, TABLE II presents the 
descriptive data referred to in the analysis. 



TABLE II 



MEAN SCIENCE ACHIEVEMENT PRESENTED BY 
UNIT, SEX, PROGRAM AND ACHIEVEMENT LEVEL 



Light Unit 



Boys 



• 


.T 


1 


2 


3 


T 


1 


2 


3 


Skinnerian 


97 


99 


96 


96 


92 


97 


93 


88 


Ruleg 


90 


85 


91 


92 


92 


96 


92 


89 


Read Skinnerian 


82 


80 


81 


83 


82 


87 


81 


80 


Read Ruleg 


80 


82 


82 


78 


82 


80 


88 


78 








Heat 


Unit 












Boys 








Girls 






Skinnerian 


80 


88 


75 


84 


71 


75 


68 


71 


Ruleg 


68 


64 


65 


71 


72 


81 


71 


69 


Read Skinnerian 


66 


73 


63 


65 


65 


74 


66 


59 


Read Ruleg 


71 


79 


68 


68 


59 


70 


55 


56 
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T • total achievement 

1 * level 1 - knowledge 

2 * level 2 - example and application 

3 * level 3 - relationship 



(TABLE II 5 cont.) 



HEAT 

IQ 



Skinnerian 111 
Ruleg 109 
Head Skinnerian 110 
Read Ruleg ill 



LIGHT 

Skinnerian 121 
Ruleg 109 
Read Skinnerian 112 
Read Ruleg 109 



HEAT 







IQ 


Skinnerian 


Boys 


112 




Girls 


110 


Ruleg 


Boys 


106 




Girls 


111 


Read Skinnerian 


Boys 


107 




Girls 


112 


Read Ruleg 


Boys 


107 




Girls 


115 




Error Rate 

8.14 

23.4 



6.83 

14.75 



Error Rate 

7.5 

9.0 

3i.u 

16.2 
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(TABLE II - Cont.) 


LIOHT 


IQ 


Error Rate 


Skinnerian 


Boys 


125 


5.33 




Olrls 


118 


8.33 


Ruleg 


Boys 


105 


21.0 


Olrls 


113 


8.5 


Read Skinnerian 


Boys 


110 


~ — 




Olrls 


113 


— — 


Read Ruleg 


Boys 


108 


— — 




Olrls 


109 


• - 



3. How do learner characteristics Influence achievement 
levels for each of the programed versions treated 
separately? 

The relationship between intelligence and sex with achieve- 
ment levels for each of the four proTrcji versions is presented 
In TABLES X, XI, Appendix D. xhe findings are as follows: 

A. Intelligence influences the science achievement at 
all levels for the Skinner type program. This relationship Is 
also evidenced In the Read Skinner approach. 

B. Sex Is a significant variable (level 3) for the Ruleg 
version. Intelligence Is not a significant predictor variable 
for this method. Rather, the previous analysis suggested 
error rate as the critical variable. 

C. The textual versions Indicate no significant relation- 
ship between sex and achievement levels. 

;V T . 

“ * . /' ■’ 

A summary of findings Indicates: (1) Developmental 

versions (Ruleg and Skinner) are generally superior to textual 
approaches. The finding holds for all levels of the lights 
unit and for level J of the heat unit; (2) Intelligence Is 
a significant variable for the Skinnerian presentation and for 
all achievement levels .(when error rate is not controlled). 

Sex appears as slgnlflcanc at level 3; (3) Program quality 
Is crucial in determining program differences and Identifying 
related learner characteristics. 

Skinner vs. Ruleg 

$. What are differences between the two developmental 
(Skinner vs. Ruleg) programed versions? 




^ ► v ~ 










An analysis of the two programed versions Included the 
treatment effects (Xc-Xg), unit differences (X Q -X 10 ), 
Intelligence (Xj*), sex°(X 12 -X.-) and error rate CJUj.) as 
predictor variables of the '‘'achievement levels. The ±M analysis 
revealed the following Instances of a significant Influence 
(Appendix E, TABLE XII): 

* 

A. No differences were found between the two 
programed versions. 



B. Unit differences favoring the light program were 
significant at all achievement levels. 

C. Intelligence was significant at total, 1, and 2 
achievement levels, did not represent a significant effect 
at the highest achievement level (3). 

i • 

D. Sex and error rate were significant variables at 

level" 3. 

‘ ‘ -s. 1 



in Summary, although TABLES I and II show a greater mean 
achievement associated with the Skinnerian version, this 
treatment was not found to significantly differ from the 
Ruleg version when Intelligence, sex,, unit differences and 
error rate are controlled. 

As one would expect from the previous analysis of the 
four program versions, the light unit Is generally superior 
at each achievement level. At level 3, Intelligence Is no 
longer a significant Influence. Instead sex and error rate 
are significant predictor variables at this level. 

5'' What are the effects related to the two programed 
versions, when heat and light units are analyzed 
separately? 

The predictor variables are defined as Intelligence, 
sex, error rate, and program version. The statements below 
are summarized from TABLES XIII and XIV, Appendix F. 

A. The significant predictor variables -light unit- 

are: (1) error rate significantly Influences achievement 

level 1 and total; (2) I.Q. Influences level 2; (3) no 
variable Influences level 3* 

B. The significant predictor variables -heat unlt- 
are: (1) Intelligence Influences achievement levels 1 

and 2 (and total) : (2) no variable significantly Influences 
at level 3* 
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In s ummar y , a. comparison of the two developmental program 
versions indicates no significantly greater achievement related 
to the Skinner-type, although the general mean tendency favors 
this presentation. The findings suggest that for a program of 
lesser effectiveness (heat), intelligence has a relatively 
greater influence on achievement than for a ;mit which results 
in greater student achievement (light). Where intelligence is 
not observed to be a significant variable, error rate and/or 
sex appear as significant variables in a more effective program. 

The relationship between the predictor variables (program 
version, intelligence, sex, and error rate) and levelSgOf 
achievement is lowest at level 3 (R heat unit; R *.29, 

light unit). TABLES XIII and XIV give values of R at all levels. 
Because of the relatively low R 2 , the influence of achievement 
levels was observed by including achievement levels 1 and 2 
as predictors of achievement level 3. The regression analysis 
is presented in TABLE XV, Appendix G. It can be observed from 
TABLE XV that the value of R 2 moves from .29 to .65 (heat unit) 
and to .36 (light unit) when achievement levels 1 and 2 are 
added as predictor variables • The heat unit is influenced 
to a greater extent by previous achievement levels, where 
level 1 is a significant predictor variable. 



II. AUTOMATED READABILITY ANALYSIS 



The variables of the revised version of the readability 
formula (MODEL II) are presented below. 



Category I: Frame Characteristics : Step Size, Vocabulary 

Difficulty. 



X : Average nun :ar of paragraphs per frame per sample. 

Average r mber of sentences per paragraph per sample. 



X^: Average number of words per sentence per sample. 

Xjjt Average number of letters per word per sample. 



X • Average number of words per sample which were outside a 
5 standard text, i.e. Thorndike’s list of 6,000 words. This 

is a measure of dif *i cult $ words. A dictionary of any 
kind can be generated to determine the frequency and 
difficulty of a word. 



Category II: Response Characte ristics : Overt and Covert 

Responses, Multiple Choice and Constructed 
Responses, Response Relevancy. 

X^: Per cent of frames that are response frames per sample. 
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Xg : Per cent of frames with math and scientific terms among 

alternative responses per sample. 

X^: Per cent of frames with word in frame identical to 

* response alternative . 

Xiq : Per cent of frames with yes-no or true-false response 

AU alternative . 

X..: Average number of frames in which response is placed 

within frame (as opposed to last word). 

X. p* Per cent of frames with one word or phrase (1-3) as 
response alternative. 



Category III: Content Presentation, Organization, and Overlap 

or Repetition 

Density of Mathematical and Scientific Terms 

X,-: Average number of mathematical and scientific words 

* (terms) per sample. 

X l4 : Average number of letters per technical word. 

X 15 : % of frames in which the same technical term appears more 

A ^ than once . 

X-g Average nuniber of frames in which the same technical word 
10 appears consecutively. 

X-_: Average number of consecutive Instances of technical 

A ' word overlap. 

Category IV: Sentence Structure or Syntax (Average number of 

sentences containing the following kinds of words) 

X l8 : Integrators: this, that, it j 

X,qI Signals: because, but, although, as, since, when, then, 

A * next, consequently, hosier, either 

X2Q : Comparative : than 

Xgi : Average number of words per sentence, X^g sentence type 

Xg2 ; Average number of words per sentence, X^g and Xgg 

Xgg: Average number of technical words per sentence, X^g 

*24: Average number of technical words per sentence, X^g and X^q 



I 

, ERIC 

L=! 



-46- 






: Science unit 

Y-: The criterion variable: average number of errors per 

x sample. 

Example of Model II 

As an example of Model II consider the following data 
which Include the first two frames sampled from 195 frames and 
45 pages of a computer-based program system units (Skinnerian 
and linear) on light. 



EXAMPLE MODEL II 

Data — — (pag<) 

1. This Is the story of Rocky. Rocky 
was a little boy who lived many, many years ago. He lived 
frith his family In a cave. Rocky could not play after the 
sun set In the evening because there was no light. The sun 
gave him light as we have sunlight today. The sun Is one source 
of our . 

a. sound b. music c. light d. water 

tfe also get light at night from the 
stars. The stars are another source of . 

a. sound b. heat c. music d. light 

The automated variable determination for this sample range 



Is: 



AUTOMATED VARIABLE DETERMINATION 



P aragraphs Per Frame 1.00 
Sentences Per Paragraph 4.00 
Words Per Sentence 9.13 
Letters Per Word 3.88 
Percent Words Not in Suds Dictionary l4.8l 
Frames In the Sample 2.00 
Percent Response Frames 100.00 
Percent Frames with Related Terminology 100.00 



Percent Frames With Word(s) Identical to Resp. Alternative 

50.00 
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Percent Frames With Boolean Resp. Alternative O.CX) 

* '■*“ "** *' V* S • 't-t 

Percent Response Frames, Response Not Last 0.00 

Percent Frames With Word/Phrase Response Alternatlve(s) 100.00 
Percent Related Terms 23*46 

Letters Per Related Term 4.21 

Percent Frames With Multiple Occurrences of Related Terms 

50.00 

Cases of Framewise Overlap 2.00 

Cases of Consecutive Overlap Series 2.00 

Percent Integration Sentences 12.50 

Percent Signal Sentences 25.00 

Percent Comparator Sentences 0.00 

Words Per Integrator Sentence 6.00 

Words Per Signal/Comparator Sentence 13.00 

Related Words Per Integrator Sentence 1.00 

Related Words Per Signal/Comparator Sentence 2.00 



The regression equation which was established by using average 
student error rates and the 25 variables automatically as Inde- 
pendent variables (on samples of 2 to 3 frames) and doing this 
for the entire light unit as follows: 

Y t =0.00X3^ + 0.53634252X2 + 0.42380359X3 + 0.24834082X4 

-0.06597135X5 + O.25568OIIX5 

=0.054l7647X 7 + 0.l6048602X 8 - 0.12390994^ + 0.082l4095X 1Q 

+ 0 . 16147853 X!! + 0.08054411X22 

=-0.14198652X23 - 0.32043226X24 - 0.01439624X25 ' * 

- 0.13903314X25 + 0.1754i607X 17 

=0. 11554272X^0 +0.28266357X3^ -0.13069124X2Q 

- 0.19389049X22 - 0.29775394X22 - 0.15400293X23 

+ 0.32268684x 2 4 

ERIC 
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In this analysis about 20,000 words and 45 pages of text 

were analyzed in seconds; The (regression) equation related 

the independent variables to the criterion variable (error rate) 

with a multiple correlation coefficient (R = *84), as high as 

or higher than any reported in the literature where less data 

and more laborious non-automated techniques have been used. 

This equation accounts for 70# of all the variability among 

individuals (obtained by squaring the multiple correlation 

coefficient ( ,84 2 » .70) error rates. This figure is 220# 

better than our original prototype. Model I, (R - .573* ( »5?3)^ 
.70 

= .32) 732 =2.2) and yields better results than attempts 
to use programing variables such as overt, covert, step size, 
etc., which have not given any clear-cut predictability or 
reliability and whose results have been inconclusive. As is 
seen by the magnitude of the standard regression weights in 
our (above) equation, all o $ the four categories of variables 
are represented and important; three of these categories in- 
troduce variables synthesized from 50 or more learning studies 
which have never before been used in readability work. 



A pplication of Readability Analysis to Skinnerian and Ruleg Programs 

The readability analysis was applied to the Skinnerian and 
Ruleg programs, heat unit. The automated text tallies are 
given In TALLSS XVI and XVII, Appendix H. 

The listing suggests comparability between the two programs 
In step size and general vocabulary difficulty. As would be 
expected, the Skinner version contains more frames (2oo; than 
does the Ruleg program (155). 



Both programs have a similar number of response frames, 
although the Skinnerian version contains about 10 per cent more 
frames with technical terminology as a response alternative. 

This count supports the greater emphasis placed on discrimination 
in the Skinner-type program. However, the lesser percent 
frames with words in frame identical to the response alternative 
found in the Skinner program may not be consistent with tne 
greater number of prompts usually considered necessary to ac-ieve 
discrimination in a Skinner-type program. 

The content presentation variables reveal comparability 
between the versions in technical vocabulary. However, the 
Skinner program contains a greater percentage of multiple 
occurrences of technical terminology and a larger 2 m nn *,* 

cases of overlap. These results are consistent with tne Skinner- 

- principle of shaping behavior. 

The listing of the syntax variables indicate that the 
percent of signal and comparative sentences is greater for the 
Ruleg program. In addition, the number of technical words in 
the integrator, signal and comparative sentence structures is 
relatively greater in the Ruleg program. These data counts 
suggest the greater use of a complex sentence structure may be 
involved in a Ruleg program which emphasizes the stating or 
rules and applications. 



The standard weights associated with the 25 readability 
variables, Ruleg and Skinner programs, are presented in tables 
XVIII, Appendix I. 



The total error rate was lower for the Skinner program 
(3.14) than for the Ruleg version (23.4): hence, a re -atively 
lower R 5 is observed for the Skinner program. The contribution 
of variables, 1, 3, 4, frame characteristics, the number or 
integrator sentences, variable 18, and number of words con- 
tained in this sentence type, variable 21, to er *° r 
consistent with the Skinnerian principles of small step size 
and shaping behavior. It is quite possible that the use of 
•’this, that, it 3 (integrator sentences) created ambiguities, 
particularly in the Skinner approach. 
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In contrast, the number of signal and comparative sentences 
and the number of words and technical terms contained In these 
sentence types Is one major source of learning difficulty In 
the Ruleg program. A second source of difficulty Indicated by 
the standard weights Is the need for more steps and overlapping 
(variables 17 and 15)* General vocabulary difficulty and tech- 
nical word length difficulty (variables 5 and 14) substantiate 
the suggested need for more frames and greater overlap in the 
Introduction of concepts, as well as in the presentation of rela- 
tionships in signal and comparative sentence types. Reduction 
of the complexity of sentence structure (within the constraints 
of the Ruleg programing technique) is also suggested by the 
analysis . 



The branch effectiveness was evaluated by a multivariate 
analysis ( ) . The three branching conditions are defined as : 

Adaptation 1. Knowledge of - Lexical Component 

Adaptation 2. Comprehension of - Structural Component/ 



The three treatments were (1) no branching condition, 

(2) comprehension branches - adaptation 2 or 3, (3) knowledge 

and comprehension - adaptation 1 In combination with either 
adaptation 2 or 3* Predictor variables were defined as: 

Xg - Xjj « achievement at levels 1, 2, 3, respectively 

X^ * error rate 

Xg • time (in minutes) 

X 1Q * intelligence 



The criterion variable was defined as science achievement 
of current items on subtests): 



III . BRANCHING CONDITIONS 



Sentence 

Adaptation 3* Comprehension of - Structural Component/ 

Topical 





• total achievement 



• achievement at Level 1 - knowledge 



® achievement at Level 2 - example and application 



* achievement at Level 3 - relationship 
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• .-The results of the multivariate analyses are summarized in 
TABLE-, XIX, Appendix J. Time, intelligence and sex are defined 
as control variables -at: all achievement levels. In addition, 
levels of achievement and error rate are included as controls 
when total achievement is the criterion variable. The findings 
are as follows: 

1. Branching adaptations significantly influence science 
achievement at all achievement levels . 

2. Branching adaptations contribute to total science 
achievement beyond that accounted for by levels of achievement 
(knowledge, application, relationship) and error rate in combina- 
tion with knowledge attainment . 

TABLE III presents the mean science achievement, error 
rate, time (in minutes), and intelligence for each of the three 

branching treatments. ’ - 

* ( \ 

TABLE III 



Mean Science -Achievement, Error Rate, Time, and 
Intelligence for the Three Branching Treatments 



Treatment 1 
No Branch 



Total Achievement 70 
Level 1 - Knowledge 82 
Level 2 - Application 67 
Level 3 - Relationship 63 
Error rate .21 
Time 2 §5 
Intelligence III 



Treatment 2 Treatment 3 

Comprehension Knowledge & 

Branches Comprehension 

Branches 

77 -84 i 

89 93 - 

. 71 80 > - - 

74 81 

14 ; - f- 10 

. „ 319 \Oi:T 273 

106 115 



It can be observed from TABLE III that the descriptive data 
are consistent with the analysis and with the branching rationale. 
Although direct comparisons are not possible because of differ- 
ences in mean intelligence (intelligence, sex and time were 
controlled in the analysis), total achievement increases from 
the no . branching condition to the presentation of both knowledge 
and comprehension branches. Comprehension (Levels 2 and 3) is 
improved with the introduction of comprehension branches, and 
knowledge attainment (Level 1) is highest rwhen the knowledge 
branching condition is introduced under Treatment 3* Error rate 
likewise drops with the addition of branching conditions. 

J;:. summary, the branching rationale holds promise as an 
effective means to remediate the reading difficulty of programed 
texts. With the inclusion of branches, science achievement is 
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improved, error rate drops, and the relationship between intelli- 
gence and achievement is reduced. That is, the correlations 
between intelligence and total achievement are r » .6*1, r * .52, 
r * .48 for treatments 1, 2, 3, respectively. 

Application to Readability Analysis 

The content of the heat unit into which the branching 
versions were introduced was analyzed using the automated reada- 
bility analysis. The standard weights associated with the 
readability variables are presented in Appendix K, TABLE XX. 

The between the contextual variables and error rate equals .77. 

In this analysis, the frame number was included to ascertain 
the influence of sequencing on error rate, and to determine if 
the assumption of a logical presentation was met. The variable, 
frame number, is the primary contributer to error rate; i.e., 
students progress through the unit with an increasing number of 
errors, A positive relationship is observed between frame 
number and number of technical terms (r ° .18), between frame 
number and technical word overlap (r ,s .47), and between frame 
number and technical words included in integrator and signal/com- 
parative sentences (r * .26; r ■ .33). 

The greater attainment of knowledge relative to the com- 
prehension achievement levels and to the no branching condition 
(TABLE III) is evidenced by the insignificant contribution of 
technical terminology to error rate vs. the significant contri- 
bution of technical words contained in signal/comparative 
sentence structures. Further study with a larger sample sise 
would allow one to ascertain the extent to which comprehension 
branches reduce the reading difficulty of sentence types, 
integrator, signal/comparatives . It is possible that the 
adaptation, comprehension-sentence structure is more appropriate 
for the signal/comparative sentence types and that the compre- 
hension adaptation-topical, is more effective in reducing the 
aniblgulty previously observed with integrator sentence structure. 



IV. FACTOR ANALYSIS OF READABILITY MODEL 

To facilitate interpretation of the variables in the model, 
a principle components Factor analysis of the preceeding data 
(Model presented in C "E XVIII Appendix I was completed with 
variance rotation(43) and yielded four factors. These factors 
and their loadings Are presented in TABLE IV • Only those 15 
variables which gave principal loadings on one of the four 
factors are Included in the table. 
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TABLE IV 



Variable 


Factor 


Factor 


Factor 


Factor 


No.* 


I 


II 


III 


IV 


16 


* 0.83311 


0.061(5 


0.0018 


-0.1602 


13 


* 0.7963 


-0.0913 


0.0348 


-0.2367 


17 


* 0.7907 


0.1841) 


0.1883 


-0.0729 


8 


* 0.5833 


-0.1086 


-0.1337 


0.2304 


3 


- 0.0315 


*0.7949 


0.4070 


-0.1340 


22 


-0.2699 


*0.6990 


-0.1796 


-0.3339 


24 


0.1809 


*0.6923 


-0.1329 


-0.0560 


23 


0.0213 


*0.6700 


- 0.0345 


0.1435 


21 


-0.3888 


*0.6556 


-0.0887 


-0.2006 


2 


0.2832 


0.0098 


*0.7528 


-0.1040 


6 


0.1068 


-0.2570 


*-0.7198 


-0.3756 


7 


0.0066 


0.1527 


*-0.6951 


0.1406 


12 


0.1269 


0.1455 


*-0.6450 


-0.1623 


4 


-0 .124*1 


-0.0892 


0.0581 


*0.7610 


5 


-0.3021 


-0.2761 


-0.1973 


*0.7321 



•See Section II, Automated Readability Analysis for 
variable identification. 



Inspection of theee loadings indicates that Factor I consists 
primarily of variables from category 3, content presentation 
and organization; Factor II from category 4, sentence structure 
or syntax; Factor III from category 2, response characteristics, 
and Factor IV from category 1, frame characteristics. These 
loadings suggest that the original classification of reada- 
bility variables was appropriate. 



V. SUMMARY OF FINDINGS 

1. The developmental presentations, Skinnerian and Ruleg, 
were generally superior to the textual counterparts at the three 
achievement levels, knowledge, comprehension, and relationships. 

A. The Skinnerian version was found to be generally more 
effective than the Ruleg presentation. This finding is qualified 
by the greater error rate associated with the Ruleg version, which 
may be attributed either to the method per se or to the prcgramers* 

ability. 



B. Intelligence is more closely related to both develop- 
mental and textual presentations of the Skinner type. 

2. The two units, heat andli&xt, differed in program quality. 
The light unit was observed to be more effective at all achievement 
levels. Program quality was observed to be a critical variable in 
attempting to establish the superiority of a program version and in 



o 
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identifying significant learner characteristics. Where intelligence 
was not a significant variable , error rate and sex appeared as 
significant. In addition, achievement at the lowest conceptual 
level appeared as significant to the attainment of relationship 
objectives in the lower quality program. 

3. Application of the automated readability analysis to the 
developmental versions, Skinnerian and Ruleg, indicated comparability 
between the two programs in step size and general vocabulary 
difficulty • 



A. Greater emphasis on shaping behavior and discrimination 
was evidenced 4 *i the Skinner program by the greater number of frames 
(Skinner - 288; Ruleg - 155), the greater number of cases of overlap, 
and about 10 percent more frames with technical terminology as a 
response alternative. However, a lesser percent of frames with a 
word in the frame identical to the response alternative may not be 
considered with the greater number of prompts usually considered 
necessary to achieve, discrimination in a Skinner-type program. 

B. A listing of the syntax variables indicated that 
application of rules was applied to a greater extent in the Ruleg 
version. This program contained a <*reater percent of signal and com- 
parative sentences. In addition, number of technical words in 
the Integrator and comparative sentence structures is relatively 
greater in the Ruleg program. These variables comprised one major 
source of learning difficulty in the Ruleg program. The automated 
analysis suggested the need for more frames and greater overlap in 
the introduction of concepts, as well as in the presentation of 
relationships in signal and comparative sentence types. 

C. The contribution of three variables under frame 
characteristics and the number of Integrator sentences to error 
rate ir. the Skinnerian program is consistent with the principles 

of small step size and shaping behavior. It is quite possible that 
the use of "this, that, it" (integrator sentence structure) created 
ambiguities, particularly with the Skinner approach. 

4. The automated readability analysis proved successful. 

In Model I seventeen independent variables were automatically 
determined and in Model II the number of Independent variables was 
extended to twenty-five (See Appendix L for prototype of computer 
program) • 



A. On samples of 2 to 3 frames, 45 pagee of text, the 
25 independent variables were related to error rate with R * .84. 
Thus, the automated analysis accounts for 70 percent of all of the 
variability among individuals' error rates. 

B. A factor analysis of the readability analysis validates 
the four categories of Independent variables: frame characteristics, 
response variables, content presentation and organization, and the 
syntax classification. 



•w- 
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5. The branching rationale, (1) Knowledge of - Lexical 
Component, (2) Comprehension of - Structural Component /Sentence, 

(3) Comprehension of - Structural Component /Topi cal, holds promise 
as an effective means to individualise the program sequence. 

A. When branching conditions were classified into 
treatment groups, (1) no branching condition, (2) comprehension 
branches, and (3) knowledge and comprehension branches, the 
branching adaptations significantly influenced science achieve- 
ment at all achievement levels. The mean achievement favored 
groups 3, 2, and 1, respectively. 

B. The branching adaptations contributed to total science 
achievement beyond that accounted for by levels of achievement, 
error rate, time, intelligence, and sex. 

C. The branching program met the assumptions of good 
sequential development. Significant and positive relationships 
were observed between frame number and error rate, between frame 
number and number of technical terms, and between frame number and 
technical word overlap. 

D. The application of the automated readability analysis 
to the branching program in which the individual contribution of 
the independent variables to error rate could be observed was con- 
sistent with the mean comparisons between treatment groups. 
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Chapter IV 

CONCLUSIONS AND RECOMMENDATIONS 



The major accomplishments of the study are summarized below. 

Pour program Versions 

The evaluation of the four program versions , (1) Developmental - 
Skinner-type, (2) Developmental - Ruleg, ( 3 ) textual counterpart of 
the Skinner-type, and (4) textual counterpart of the Ruleg presenta- 
tion, suggested that the developmental presentations were more 
effective at the three achievement levels (knowledge, comprehension, 
relationships) than the textual presentations. This finding was 
was evidenced for all Intelligence levels. 

Program quality was critical In the evaluation of the four 
versions. Differences In error rate between the two units, heat 
and light, were accompanied by a change In the relationship of In- 
telligence, error rate, sex, and lower achievement levels to the 
criterion, science achievement. The error rate of the Ruleg 
presentation was relatively higher than that of the Skinner version. 
Although the mean achievement favored the Skinnerian approach, a 
comparison of the two developmental presentations must be qualified 
to allow for differences In a programmer's writing style before the 
superiority of any one method can be established. The application 
of the automated analysis Indicated that the syntax category of 
lr dependent readability variables was crucial to the success of 
bot 1 the Skinner and Ruleg programs. Comparative and signal sen- 
tences were more frequent In the Ruleg presentation, and contributed 
to learning difficulty for this version, while Integrator sentences 
(this, that. It) were related to error rate In the Skinner approach. 

The evaluation of the four program versions recommends a con- 
sideration of learner and task variables. However, the findings 
suggest that before comparisons can be meaningful and generallzable, 
the program structure must be objectively defined and the Influence 
of contextual variables accounted for. Such a contextual analysis 
of programed materials appears even mo~e Important, since the 
findings of this study favor the development presentations over 
the textual versions. The use of the computer for management, as 
opposed to Instruction, can only be given qualified support. 

Automated Readability Model 

Several features of the automated readability analysis offer 
specific advantages for the use of this model In the management of 
programed materials development. One outstanding feature of this 
model Is Its flexibility. At Its present stage of development 25 
variables are used to determine reading difficulty. In principle 
an unlimited number of variables can be generated. The automated 
system allows thoroughness and completeness In using entire texts, 
a marked contrast to past readability formulas. 
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Another important feature of the model Is the ease with which 
it can be used. Natural language Inputs from a variety of sources 
such as magnetic tapes, data cards and disc storage can be directly 
used. Program translations that are costly and time-consuming are 
unnecessary. 

Past readability formulas were ready-made for use by laymen and 
other non-computationally oriented persons, resulting in restrictions 
on the clerical effort and computational skill needed to apply the 
formula. Samples of textual materials rather than entire texts were 
used. These samples were often in adequate and not representative 
of the materials they were taken from. Both limitations were 
necessary because man, not a machine, was doing the work. 

The automated feature and related analysis offer specific 
advantages to the production of programed materials (1) either 
directly, in the writing, revision and evaluation of materials; 
and (2) or through experimentation, in program definition and evalua- J 
tion of the relative influence of methods on program structure. 
Applications of the automated model to the management of materials 
development are presented below. 

1. The automated procedures and regression analysis is adapted 
to and provides an objective measure of the reading difficulty of 
programed materials. The procedures developed by the study are 
applicable to a general analysis of self-instructional materials and 
are specifically adapted to an evaluation of upper elementary school 
science. Independent variables automatically determined by the 
computer program are: * 

Categ o ry I: Frame Characteristics . Six variables which 

measure paragraph, sentence and word length, vocabulary difficulty 
and number of frames per sample. 

Category II: Response Characteristics . Six variables which 

^^cribe the number of response frames, the nature and relevance 
of the response. 

Category III: Content Presentation a nd Organization . Five 

variables which measure density of mathematical and scientific 
terms, overlap and repetition. 

Category IV: Sentence Structure or Syntax . Seven variables 

which identify three sentence types, the average number of words 
and technical words per sentence type. The remaining Independent 
variable identifies the science unit. 

Criterion Variable . Average number of wrrors per sample. 

2. The readability formula predicts student error rate, thereby 
providing a measure of the effectiveness of the materials without 
involving the expense and time in field or laboratory testing. In 

an analysis of 45 pages of text taken from an upper elementary science 




unit, the regression equation related the independent variables to the 
criterion variable (error rate) with a multiple correlation coefficier|t 
(r s .84), as high as or higher than any reported in the literature 
where less data a n ^ more laborious non-automated techniques have 
been used. Application of the automated analysis to two programed 
units contained in the BSCS Special materials accounted for 86 and o4 
percent of the variability among individuals* error rates (59)* 



3. The automated feature makes possible (or feasible) the 
analysis of the entire text at a substantially lower cost and time 
exoenditure than would be required of most manually conducted samples. 
An analysis of about 20,000 words was completed in seconds. 



4. Revision procedures are more efficient: (a) the time lag 

created by field testing and manual counts is reduced; (b) the 
analysis of the entire text allows the identification of differences 
among the writing practices of programers, the differences among 
units, hierarchies, or sequential organization; (c) the summary data 
and analysis suggest revisions specific to the program context, 
hence avoiding the trial and error often associated with those 
program modifications, which* are. unrelated to student performance 
or relevant contextual variables. 



5. The automatic count and listing and regression analysis 
objectively defines and relates slgnlf leant frame, response content 
presentation, organization, and syntax variables to error rate, 
thus providing a comprehensive definition of program structure. 
Contextual differences between Skinnerian and Ruleg progaams were 
identified, and sources of learning difficulty were isolated after 
the introduction of three branching conditions. The automated 
analysis makes possible the determination of why a branching condition 
is successful, and objectively describes similarities and differences 
between programing techniques. 

6. The independent variables were synthesized from an analysis 
of more than 200 previous studies in readabl e *.ty and an analysis of 
over 50 studies of programed learning. A principle components factor 
analysis was completed with varimax rotation and yielded four factors. 
The loadings suggest that the fourfold classification of independent 
variables was appropriate' 



The branching rationale can be described as a threefold 
hierarchy, with each level or branching condition more difficult, 
complex and abstract than the previous lie vel. The first level deals 
with technical terminology; students at this level dequire a know- 
ledge of concepts. At the second level an analysis is made of 
sentence length and structure. A topical presentation of preview 
describes the third branching condition. These three branch 
conditions can be summarized as (1) Knowledge of - Lexical Component, 
(2) Comprehension of - Structural Component /Sentence , and (3) 
Comprehension of - Structural Component/Topical. 



V. *t* 
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Initial evaluation of the three branching conditions indicated 
that the three program adaptations decreased the level of reading 
diffic* lty so that a student understood the materials that he 
previously found perplexing. The findings suggest that the 
branching rationale will enable one to determine when readability 
increases or deereases for students of varying ability and knowledge. 

In summary, the ant ornate a future and related analysis appear 
to f f er specific advantages for the analysis and production of 
self- instructional materials in three principal ways: first, directly, 

in the development of materials by managing their writing, revision, 
and evaluation; second, to develop and analyze diverse programs; 
third, managing instruction by means of branching conditions which 
take into account variation in learner characteristics. 
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SUMMARY 

Statement of Purpose and Objectives 

Despite technological advances, the production of self- 
instructional materials remains costly and Inefficient. The 
expenditures of time and money invested In testing and analysis has 
not resulted in a set of principles of programed instruction to guide 
in the writing and revision of programed materials . Nor has experi- 
mentation in programed instruction agreed on the influence of a 
programing variable or a rationale to select and adapt a given 
strategy to the programed context. The research reported in this 
project focuses on the effective development, evaluation, and 
individualization of programed materials. 

Phase I 

Efforts to develop successful programing methods have led to the 
wide variation in the application of the principles of learning to 
materials development. However, program assessment studies have not 
generated a set of principles to guide in the selection of a 
programing technique. Learner characteristics and task variables 
are freq ue ntly not considered in identifying the appropriate pro- 
graming technique* Phase I of the research activity responds to 
this need by an evaluation of four programing methods which accounts 
for individual differences and variation in conceptual attainment. 

The study did not favor any particular programing method, but 
served as a pilot approach by selecting four programing methods 
from among the many techniques, and by relating the effectiveness 
of the method of presentation to learner characteristics, program 
quality apd to the conceptual level of the behavioral objective. 

The four Methods selected were (1) developmental - Skinnerian, 

(2) developmental - RULED, (3) textual counterpart of the Skinner 
type, and (4) textual counterpart of the RULE r presentation. 

Phase II 

Research studies tend to agree that reading difficulty is an 
important criterion for selecting textual materials; and, as such, 
textbook selection is an exceedingly important and difficult task. 
Reading difficulty is even more crucial to the students* success 
in using programed materials. Despite the apparent relationship 
between reading difficulty and self-instructional materials, 
readability has largely been ignored in the study of programed 
learning. The significance of the reading factor and the absence 
of a formula adapted to the programed approach prompted the research 
activity of Phase II of the reported research. The major objective 
of this phase of activity was the development of an automated analysis 
to manage programed materials development which includes a synthesis 
of both programing and readability variables. 

Phase II of the study identifies 25 independent readability 
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variables which are descriptive of frame characteristics, response 
variables, content presentation and organization, and syntax 
variables. The automated procedures sort, locate, and accumulate 
a count and listing of the independent variables from the context 
of the programed materials. After a printout of the variable coun s 
and word listings is obtained, the independent variables are related 
to student error rate (validated against student ability and achieve- 
ncnt measures). The readability formula is described by an equation 
of the best predictors of error rate. Thus, descriptive Q ata are 
presented for all variables, those readability variables signifi- 
cantly related to student performance are identified, and the nature 
of the relationship is established. 



Phase III 

The definition and evaluation of three remedial branching 
conditions derived from the automated readability model is the focus 
of Phase III. The branching rationale can be described as a jhree- 
fold hierarchy, with each level or branching condition more difficult, 
-onrolex and abstract than the previous level. The first level 
d^ais with technical terminology; students at this level acquire 
a knowledge of concepts. At the second level an analysis is made 
of sentence length and structure. A topical presentation or preview 
describes the third branching condition. 

The objectives of each phase of research activity are outlined 

below. 



Phase I. The evaluation of the four program versions (Skinner- 
type, "HulegT and the textual counterparts of each developmental 
presentation) was carried out to identify the technique (s) most 
effective for the attainment of science achievement at three levels, 
’knowledge, application, and relationships. The analysis focused on 
the following two objectives: 

1. VJhat is the relationship between the four programing versions 
and the science achievement of fourth and fifth grade students? 

2. What are the learning conditions which describe the relative 
success of each of the four presentations? 

Learning conditions are defined as learner characteristics 
(intelligence *did sex) and program quality (error rate and science 
achievement). / 

i 

Phase II. I The following objectives were specified in the 
development arid irp Cementation of the automated readability analysis. 

1. Abstract and classify from readability and programed learning 
r'ndies those independent contextual variables to be automatically 
determined by the readability analysis. Validate the independent 
variable classifications. 
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2. Develop a computer program which sorts, locates, and 
accumulates a count and listing of the independent variables from 
the context of the programed materials. 

3. Implement the readability analysis by relating the inde- 
pendent variables to error rate. The readability formula is described . 
by an equation of the best predictors of error rate. 

4. Apply the automated readability analysis to the Skinnerian 
and Ruleg programs (Phase I) and to the programed unit which incor- 
porates the three branch conditions (Phase III) in order to objectively 
define the program structure and to distinguish similarities and 
differences between programing techniques. Determine if the 
assumptions of sequencing are met. 

Phase III . A fully developed readability model requires indi- 
vidual adaptations which decrease the level of reading difficulty. 

The objectives outlined under this phase of research activity are 
as follows: 

1. Derive a branch rationale f^om the automated analysis. 

2. Evaluate the branching conditions: (1) Knowledge of - 

Lexical Component, (2) Comprehension of - Structural Component/Sentence, 
(3) Comprehension of - Structural Component/Topical. 

3. Include in the evaluation control for intelligence, sex, : 
time, error rate and science achievement levels (knowledge, applica- 
tion, relationships). 

Procedures 

Phase I . Two programed science units, heat and light, were 
each rewritten using the four presentations: 

1. Developmental - Skinner-type 

2. Developmental - Ruleg 

3 . Summary - textual counterpart of (1) 

4. Summary - textual counterpart of (2) 

The programed units were adapted to an IBM 1460 Computer System. The 
developmental branches eequired the student to respond to each frame; 
students assigned to the summary treatments read the same materials 
written in textbook form. All students received the same subtesting 
series and remedial branches on the computer, and all were engaged 
in the individual performance of simple science experiments. 

Fourth and fifth grade students, matched by ability and grade 
level, were randomly assigned to the four treatments (n*56, heat 
unit; n»48, light unit)- St’.dent characteristics were measured by 
the Lorge-Thorndike Intelligence Test and the sex of the student was 
recorded. Each subtest item was classified as follows: 

Level 1 - concept/terminology (Level 1, Bloom) 

Level 2 - example and application (Levels 2 and 3> Bloom) 

Level 3 - relationship (Levels 4 and 5> Bloom) 
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A multiple regression technique was used t j evaluate the four 
programing Versions. 

Phase II. The procedures followed the objectives outlined above. 

A flexible multi-variable— presently twenty-five— computerized 
approach to determine the readability and to guide in the writing 
and revision of self-instructional materials was developed and imple- 
mented by the Bureau of Educational Research, University of Virginia. 

Phase III . Three branching conditions were incorporated into 
a programed heat unit, adapted to an IBM 1460 computer: 

1. Knowledge of - Lexical Component 

2. Comprehension of - Structural Component/Sentence 

3. Comprehension of - Structural Component/Topical 

Forty-three fourth grade students were randomly assigned to three 
treatments: (1) no branching conditions, (2) a comprehension branch, 

(3) knowledge branch in combination with a comprehension branch. 

A multivariate analysis was used to evaluate the three branch 
conditions , where levels of achievement, error rate, time, 
intelligence, sex, and experimental treatments were defined as pre- 
dictor variables, and science achievement at three conceptual levels, 
defined as the criterion. 

Findings and Conclusions 

/ 

The evaluation of the four program versions, (1) Developmental - 
Sinner-type, (2) Developmental - Ruleg, (3) textual counterpart of 
the Skinner-type, and (4) textual counterpart of the Ruleg presentation 
suggested that the developmental presentations were more effective at 
the three achievement levels (knowledge, comprehension, relationships; 
than the textual presentations. This finding was evidenced for all 
intelligence levels. 

Program quality was critical in the evaluation of the four 
versions.. Differences in error rate between the two units, heat 
and light, were accompanied by a change in the relationship of 
intelligence, error rate, sex and lower achievement levels to the 
criterion, science achievement • The. error rate of the Ruleg pre- 
sentation was relatively higher than that of the Skinner version. 
Although the mean achievement favored the Skinnerian approach, a 
comparison of the two developments], presentations must be qualified 
to allow for differences in a programmers writing style before the 
superiority of any one method can be established. The application 
of t! * automated analysis indicated that the syntax category of 
independent readability variables was crucial to the success of both 
the Skinner and Ruleg programs. Comparative and signal sentences 
structures were more frequent in the Ruleg presentation, and con- 
tributed n Ing difficulty for this verion, while integrator 

sentences that, it ) were related to error rate in the 

Skinner app* .. 

The automated feature and related analysis offer specific 
advantages to the production of programed materials (1) either 
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directly, in the writing, revision and evaluation of materials; 
and (2) or through experimentation, in program definition and 
evaluation of the relative influence of methods on program structure. 
Applications of the automated model to the management of materials 
development are presented below. 

1. The automated procedures and regression analysis is adapted 
to and provides an objective measure of the reading, difficulty of 
programed materials. The* procedures developed by the study are 
applicable to a general analysis of self-instructional materials 
and are specifically adapted to an evaluation of upper elementary 
school science. Independent variables automatically determined 

by the computer program are: 

Cate— ry I: Frame Characteristics . Six variables which 

measure paragraph, sentence arid word length, vocabulary difficulty 
and number of frames per sample. 

Category II: Response Characteristics. Six variables which 

describe the number of response frames, the nature and relevance 
of the response. 

Category III: Content Presentation and Organization . Five 
variables v/hich measure density of mathematical and scientific 
terms, overlap and repitition. 

Category IV: Sentence Structure or -Syntax . Seven variables 

which identify three sentence types, the average number of words 
and technical words per sentence type. The remaining independent 
iriable identifies the science unit. 

Criterion Variable . Average number of errors per sample. 

2. The readability formula predicts student error rate, thereby 
providing a measure of the effectiveness of the materials without 
involving the expense and time in field or laboratory testing. 

In an analysis of 45 pages of text taken from an upper elementary 
science unit, the regression equation related the Independent 
variables to the criterion variable (error rate) with a multiple 
correlation coefficient (R = .84), as high or higher than any 
reported in the literature where less data and more laborious 
non-automat ej techniques have been used. 

3. The a;:':or.:ated feature makes possible (or fe* ^ible) the 
analysis of the er^ire text at a substantially lower >st and t ime 
expenditure than would be required of most manually condueted 
samples. An analysis of about 20,000 words was completed in seconds. 

4. Revision procedures are more efficient: (a) the time lag 

created by field testing and manual counts is reduced; (b) the 
analysis of the entire text allows the identification of differences 
among the writing practices of programers, the differences among 
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units, hierarchies , or sequential organisation; (c) the summary data 
and analysis suggest revisions specific to the program context , 
hence avoiding the trial and error often associated with those 
program modifications which are unrelated to student performance 
or relevant contextual variables. 

5* The automatic count and lilting and regression analysis 
objectively defines and relates significant frame, response 
content presentation, organization, and syntax variables to error 
rate, thus providing a comprehensive definition of program structure, 
Contextual differences between Skinnerian and Ruleg programs were 
Identified, and sources of learning difficulty were Isolated 
after the Introduction of three branching conditions. The automated, 
analysis makes possible the determination of why a branching 
condition Is successful, and objectively describes similarities 
and differences between programing techniques. 

6. The Independent variables were synthesized from an analysis 
of more than 200 previous studies In readability and an analysis of 
over 50 studies of programed learning. A principle components 
factor analysis was completed with varlmax rotation and yielded 
four factors. The loadings suggest that the fourfold classification 
of Independent variables was appropriate. 

The branching rationale, (1) Knowledge of - Lexical Component, 
(2) Comprehension of - Structural Component/Sentence, (3) Compre- 
hension of - Structural Component/Topical, holds promise as an 
effective means to Individualize the program sequence. 

1. When branching conditions were classified Into treatment 
groups, (1) no branching condition, (2) comprehension branches, 
and (3) knowledge and comprehension branches, the branching adapta- 
tions significantly Influenced science achievement at all achieve- 
ment levels. Mean science achievement favored groups 3, 2, and 1, 
respectively. 

2. The branching adaptations contributed to total science 
achievement beyond that account for by levels of achievement, 
error rate, time. Intelligence, and sex. The program met the 
assumption of good sequential development. Significant and positive 
relationships were observed between frame number and error rate, 
between frame number and number of technical terms, and between 
frame number and technical word overlap. 

3. The application of the automated readability analysis to 
the branching program Indicated that the three program adaptations 
decreased the level of reading difficulty. 



o 
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In s umma ry, the automated feature and related analysis appear 
to offer specific advantages for the analysis and production of 
self-instructi<nal materials in three principal ways: first, 

directly in the development of materials by managing their writing 
revision, and evaluation; second to develop and analyze diverse 
programs; third,' managing instruction by means of branching 
conditions which take into account variation in learner 
characteristics . 
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Key to Variables Analyzed b£ Multivariate Analysis 
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Level 2 Score 
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Heat Unit 
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Light Unit 


11 


IQ 
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Male 
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14 


Error Rate 


15 


Unit Vector 
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TABLE V 



Analysis of the Effects of Achievement Levels Attributed 
to Science Unit Differences - Developmental Versions 



TABLE VI 



Analysis of the Effects on Achievement Levels Attributed 
to Science Unit and Error Rate Differencee-Skinnerian Program 



TABLE VII 



Analysis of the Effects on Achievement Levels Attributed 
to Science Unit and Error Rate Difference s-Ruleg Program 



TABLE VIII 



Analysis of the Effects on Achievement Levels Attributed 
to Science Unit and Error Rate Differences - Read Skinnerian 

and Ruleg Versions 
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ANALYSIS OP THE EFFECTS ON ACHIEVEMENT LEVELS ATTRIBUTED 
TO SCIENCE UNIT DIFFERENCES AND ERROR RATE-RULEG VERSION 
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APPENDIX E 
TABLE XII 



Relationship between Skinner and Ruleg, Learner Characteristics, 
Unit Differences, Error Rate, and Achievement Levels 
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APPENDIX F 
TABLE XIII 

Relationship of Program* learner Characteristics, 
Error Rate to Levels of Science Achievement-LIGHT Unit 
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APPENDIX H 
TABLE XVI 

Automated Text Tallies - Skinnerian Program 

TABLE XVII 

Automated Text Tallies - Ruleg Program 
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TABLE XVI 

Automated Text Tallies- Skinnerian Program 



1.00 

1.58 

11.23 

4.35 

23.39 

288 



Paragraphs per frame 
Sentences per paragraph 
Words per sentence 
Letters per word 

Percent words not in Suds dictionary (measure of general 

vocabulary difficulty) 



Frames in the sample 



98.96 

53.47 

0.69 

2.78 

39.30 

93-40 



Percent response frames 

Percent frames with :, related n terminology* 

Percent frames with word(s) identical to resp. alternative.- 
Percent frames with "Boolean” resp. alternative (yes/no, true/false) 
Percent response frames, response not last 
Percent frames with word/phrase response alternative (s) 



16.78 Percent "related" terms 
4.27 Letters per "related" term 

97.57 Percent frames with multiple occurrences of "related" terms 
240 Cases of framewise overlap 
111 Cases of consecutive overlap series 



30.55 Percent integrator sentences 
32.09 Percent signal sentences 
5.93 Percent comparator sentences 
13.12 Words per integrator sentence 
13.62 Words per signal/comparator sentence 
1.60 "Related" words per integrator sentence 
1.8l "Related" words per signal comparator sentence 

♦"Related" terminology refers to the scientific and mathematical 
terms abstracted from behavioral objectives. 
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TABLE XVII 



Automated Text Tallies - Ruleg Program 



1 . 00 
1.73 
12.11 
4.36 
23.99 

155 



Paragraphs per frame 
Sentences per paragraph 
Words per sentence 
Letters per word 

Percent words not in Suds dictionary (measure of general 

vocabulary difficulty) 



Frames in the sample 



98.06 Percent response frames 

43.23 Percent frames with "related" terminology* 

2.58 Percent frames with word(s) identical to resp. alternative 
1.94 Percent frames with "Boolean "resp. alternative (yes/no, true/false) 
45.39 Percent response frames * response not last 
89.68 Percent frames with word/phrase response alternative (s) 



17.72 Percent "related" terms 
4.07 Letters per "related" term 

97.42 Percent frames with multiple occurrences of related terms 
172 Cases of framewise overlap 
83 Cases of consecutive overlap series 



29.10 Percent integrator sentences 
40.67 Percent signal sentences 
10.07 Percent comparator sentences 
12.77 Words per integrator sentence 
13.72 Words per signal/comparator sentence 
2.06 "Related" words per integrator sentence 
2.24 "Belated" words per signal/comparator sentence 



♦"related" terminology refers to the mathematical and scientific 
terms abstracted from behavioral objectives. 
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APPENDIX I 
TABLE XVIII 

Standard Weights Assigned to the 25 
Readability Variables - Skinnerian and P.uleg Programs 
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TABLE XVIII 



Standard Weights Assigned to the 25 Readability Variables* - 

Skinnerian and Ruleg Programs 



Skinnerian Program 



Ruleg Program 



Variable 

Number 



Standard 

Weight 



Variable 

Number 



1 


0.18277245 


1 


2 


0.00000000 


2 


l 


0.28545191 

0 . 23134649 


3 

4 


5 


0.00000000 


5 


6 

7 


0.09758769 

0.04291700 


6 

7 


8 


0.10529069 


8 


9 


-0.18017315 


9 


10 


0.08233432 


10 


11 


0.01367922 


11 


12 


0.06602573 


12 


13 


0.02303355 


13 


14 


0.07785489 


14 


15 


0.12903041 


15 


16 


0.01356095 


16 


17 


-0.13337785 


17 


18 

19 


0.19619866 

0.04090694 


18 

19 


20 


-0.06482657 


20 


21 


0.29207201 


21 


22 


0.01850725 


22 


23 


-0.02010439 


23 


2> 


0.02835074 


24 


25 


-c. 23622038 


25 



Standard 

Weight 

-0 . 00579841 
■ 0 . 00000000 
-0.16716149 
-0.14968433 
0.30950178 
-0.60336497 

0.37176365 

0.00000000 

-0.01912271 

0. 02047304 

0.17193178 

-0.26959749 

0.09922112 

-0.30187925 

-0.25758785 

-0.11935410 
-0.0606046 6 

0.15160406 

-0.23824761 

0.05828184 

0.01445657 

-0.28683381 

0.03453030 

0.74415952 

-0.1442081' 



RSQ = 0.2763 



RSQ = 0.449 



♦See Section XI, Chapter III for variable identification. 
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TABLE 



XIX 



Analysis of the Effects on Achievement Level 
Attributed to Branch Adaptations 
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APPENDIX K 
TABEE XX 

Standardized Weights Associated with the Readability 
Analysis of the -Programed Materials, Branching Version 
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TABLE XX 

Standard Weights associated with the Readability Analysis of the 

Programed Materials, Branching Version 



Variable 



Standard Weight 



Frame number 0.6727 

Paragraphs per frame 0.0000 

Sentences per paragraph -0.3411 

Words per sentence 0*3615 

Letters per word 0*0546 

Vocabulary difficulty -0.0428 

Frames per page 0,4772 

Per cent of response frames -0.1863 

Per cent of frames with technical terms * 

among response alternatives 0.0825 

Per cent of frames with word in frame identical 

to response alternative 0.2939 

Per cent of frames with yes-no ar true-false 

response alternative 0.0138 

Frames in which response is placed within sentence -0.0872 
Per«.°ent of frames with 3 words or less as 

response alternative 0.0927 

Average number of technical words 0.1502 

Letters per technical word 0.1606 

Per cent of frames in which the same technical 

term appears more than once -0.0293 

Average no. of frames in which the same technical 

word appears consecutively O.1631 

Average no. of consecutive instances of technical 

word overlap -0.2458 

Integrator sentences: this, that, it -0.0531 

Signal sentences: because, but, although, as, 

since, when, then, next, etc. 0.0495 

C omparatives sentence s : than 0.1146 

Words per integrator sentence 0.1080 

Words per signal and comparative sentences 0.2923 

Technical words per integrator sentence -0.1703 

Technical words per signal and comparative 

sentences -0.3^82 
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APPENDIX L 

Computer Program Quickly 
Prototype 
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APPENDIX L 

COMPUTER PROGRAM QUICKLY 
PROTOTYPE 



Character Set 



The acceptable characters in a "book" are all the 63 characters 
permissible in Burroughs Extended Algol. This language defines 
as ALPHA characters all characters which are either the digits zero 

through nine (0,1,2,. ,9), or letters of the alphabet 

(A,B,C, . . .Z) . All the characters which are non-ALPHA are SPECIAL 
characters . 

Of the SPECIAL characters, seven were given defined meanings 
for Quickly. These are reserved as controls and are: 

. (period), - (hyphen2» * (equal sign), / (slash), > (greater than 
sign), * (asterisk), § (blank). 

ALL SPECIAL characters which are not one of the seven controls are 
treated a 3 if they were the character blank. 

Words 

A word, to QUICKLY, Is a set of characters in a particular wrder. 
Words are delimited by blanks or central characters. 



Sentences _ _ 

Sentences usually consist of no words followed by a period, 
or any nunber of words followed by a period. Sentences are ended 
by the characters : . ^period), / (slash), * (asterisk), and 
> (greater than). 

Fr8D60 

seven different frame types are recognized. These are Response, 
Structured Response, Free Response, Non-Response, Review, Disjoint, 
and Non-Dis joint Frames. (A frame includes tally information and 
paragraphs . ) 

Paragraphs 

A paragraph is a group of sentences. The paragraph begins 
with the character / and may be terminated by any or several of 
the characters / (slash), * (asterisk), > (greater than) and 
(period). 

P age Number 

If the -.'page number/, is not present, the current page number 
will be incremented by 1. If no page number is specified fpr the 
very first page, it will be assumed 1. If a /page number^ is present 
it replaces the current page number. 
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Books (Learning Units) 

A “book" for QUICKLY is a standard ALGOL file labelled 
“ooooooo" "TEXT". It consists of one or more pages as defined 
above, punched on cards in columns 1-80. This book is terminated 
by an END OP FILE control card. 

Responses Responses are characteristic of but not restricted 

to Programed and Computer Assisted Instructional 
materials . 

Responses are groups of words bracketed by * marks. These 
word groups may also be terminated by />* control characters if 
desired. 

Output 

QUICKLY "reads” a book and sorts all words onto two lines. 

One is a list of all words bracketed as responses and the other 
is a list of non-response words. Lists are in alphabetical order. 
A word is printed, then the locations of its occurrences are 
indicated as (page number) : (frame number). 



